You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "desaikomal (via GitHub)" <gi...@apache.org> on 2023/05/18 16:14:59 UTC

[GitHub] [helix] desaikomal opened a new pull request, #2493: Waged - Hard constraint - Fix for n - n+1 issue.

desaikomal opened a new pull request, #2493:
URL: https://github.com/apache/helix/pull/2493

   ### Issues
   
   - [ ] My PR addresses the following Helix issues and references them in the PR description:
   Fixes #2468 (partial)
   ### Description
   
   - [] Here are some details about my PR, including screenshots of any UI changes:
   Waged - Constraint based algorithm has few corner cases where the hard-constraint is not satisfied. Details are documented at Wiki: https://github.com/apache/helix/wiki/WAGED-rebalancer-Hard-Constraint-Scope-Expansion
   
   Google doc: https://docs.google.com/document/d/1uxCqSU0xriUTdZIyxVoEvPMvywRgHCANatUlR0YQojk/edit#
   
   Original proposal was to throttle the ST mesages in intermediate state calculation stage. The alternate proposal is to it in BestPossibleCalcStage() but after the actual calculations are done. This is done later as we don't want to cause ping-pong effect as part of the Baseline or Partial Pipeline calculations.
   
   This change introduces a data provider, InstanceCapacity which will take into consideration pending messages on the instances.
   
   Part-2 will introduce the change where this will be used in DelayedAutoRebalancer.
   
   ### Tests
   
   - [ ] The following tests are written for this issue:
   
   (List the names of added unit/integration tests)
   TestWagedRebalancer.testInstanceCapacityProvider
   
   - The following is the result of the "mvn test" command on the appropriate module:
   
   (If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)
   
   ### Changes that Break Backward Compatibility (Optional)
   
   - My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
   
   (Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
   
   ### Documentation (Optional)
   
   - In case of new functionality, my PR adds documentation in the following wiki page:
   
   (Link the GitHub wiki you added)
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199157068


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {

Review Comment:
   fixed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199119876


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {

Review Comment:
   Based on what I have read, skip additional keys.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1200771139


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;

Review Comment:
   addressed, renamed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199157871


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {

Review Comment:
   Does this make sense?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on PR #2493:
URL: https://github.com/apache/helix/pull/2493#issuecomment-1554945992

   @jiajunwang and @junkaixue - can you please help review this? This feature is required for customer onboarding. thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199143017


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+      }
+    }
+    return true;
+  }
+
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    boolean modified = false;
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+        if (partCapacity != 0) {
+          instanceCapacity.put(key, instanceCapacity.get(key) - partCapacity);
+          modified = true;
+        }
+      }
+    }
+    if (modified) {
+      _perInstance.put(instance, instanceCapacity);
+    }

Review Comment:
   addressed your comments. thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1206834163


##########
helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java:
##########
@@ -460,4 +460,9 @@ public Map<String, ResourceAssignment> getAssignment(Set<String> resourceSet) {
 
     return currentStateAssignment;
   }
+
+
+  public Map<String, Map<Partition, Map<String, Message>>> getPendingMessages() {

Review Comment:
   Thanks for pointing that out. I wanted to get all the pending messages to process in one go. The flow is: You have capacity - all pending messages 
   this is how we start  once the assignment has been calculated.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1198152762


##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/InstanceCapacityDataProvider.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.helix.controller.dataproviders;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+
+/**
+ * An interface to provide capacity data for instance.
+ * It will consider the pending state transition and assigned partitions capacity.
+ * It will be dynamic and will always provide the available "headroom".
+ *
+ * The actual implementation will be stateful.
+ */
+public interface InstanceCapacityDataProvider {
+
+ /**
+   * Get the instance remaining capacity. 
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName);
+
+  /**
+   * Check if partition can be placed on the instance.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @return boolean - True if the partition can be placed, False otherwise
+   */
+  public boolean isInstanceCapacityAvailable(String instanceName, Map<String, Integer> partitionCapacity);
+
+  /**
+   * Reduce the available capacity by specified Partition Capacity Map.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @returns boolean - True if successfully updated partition capacity, false otherwise.
+   */
+  public boolean reduceAvailableInstanceCapacity(String instanceName, Map<String, Integer> partitionCapacity);

Review Comment:
   First of all, thanks a lot for making the quick change. This setup should work, but just to share my different opinion.
   
   My original through is that, to make this capacity provider pure and simple as possible -- It's simply getting the original instance headroom during the pipeline, it's stateless. Think about a microservice. But in this way, you are combining the data provider with a stateful cache.
   In my view, `isInstanceCapacityAvailable` and `reduceAvailableInstanceCapacity` are the responsibility of a "manager" class. That manager class will call the data provider, and maintain a cache, then reduce capacity by iterating all the resources and pending message. The manager class is stateful.



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {

Review Comment:
   Does this need to be public?



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {

Review Comment:
   As the top comment goes, this feels a little deviate from the responsibility of this class.



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {

Review Comment:
   General comment: We need to be clear whether it's a capacity tracker, or a provider.



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {

Review Comment:
   This should be in a separate class, the resource capacity provider, right?
   FYI, https://github.com/apache/helix/pull/2480/files#diff-07032ec7fdb5667dcf36e472c77dfefdd1eb2cff72feb472d2158b1df6fd9664



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1206831684


##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/InstanceCapacityDataProvider.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.helix.controller.dataproviders;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+
+/**
+ * An interface to provide capacity data for instance.
+ * It will consider the pending state transition and assigned partitions capacity.
+ * It will be dynamic and will always provide the available "headroom".
+ *
+ * The actual implementation will be stateful.
+ */
+public interface InstanceCapacityDataProvider {
+
+ /**
+   * Get the instance remaining capacity. 
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName);
+
+  /**
+   * Check if partition can be placed on the instance.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @return boolean - True if the partition can be placed, False otherwise
+   */
+  public boolean isInstanceCapacityAvailable(String instanceName, Map<String, Integer> partitionCapacity);

Review Comment:
   Will you be ok, if i keep the same name as this method is to indicate if Instance has the capacity available



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on PR #2493:
URL: https://github.com/apache/helix/pull/2493#issuecomment-1568827618

   THANKS A MILLION @junkaixue / @qqu0127 and @rahulrane50 for reviewing the change. This change has been approved by @junkaixue and @qqu0127 .
   
   Commit message: In WAGED, during N, N+1, we may not satisfy the HARD constraint resulting in failure. This is part-1 of the change where we start keeping track of instance capacity with pending messages. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1198238070


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {

Review Comment:
   we can. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1200771777


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);

Review Comment:
   For now, i will set it to -1. As we integrate with DelayedAutoRebalancer, i will make sure that this is invoked only for Waged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199158277


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {

Review Comment:
   does this answer your question?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1198237623


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {

Review Comment:
   Thanks for catching, it was accidental.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1200769990


##########
helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java:
##########
@@ -460,4 +460,9 @@ public Map<String, ResourceAssignment> getAssignment(Set<String> resourceSet) {
 
     return currentStateAssignment;
   }
+
+
+  public Map<String, Map<Partition, Map<String, Message>>> getPendingMessages() {
+    return _pendingMessageMap;

Review Comment:
   thanks. Addressed the comment. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1200818035


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }

Review Comment:
   yes, need to add more testing when as part of 2.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1200786014


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);

Review Comment:
   thanks Quincy. Updated it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1198236833


##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/InstanceCapacityDataProvider.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.helix.controller.dataproviders;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+
+/**
+ * An interface to provide capacity data for instance.
+ * It will consider the pending state transition and assigned partitions capacity.
+ * It will be dynamic and will always provide the available "headroom".
+ *
+ * The actual implementation will be stateful.
+ */
+public interface InstanceCapacityDataProvider {
+
+ /**
+   * Get the instance remaining capacity. 
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName);
+
+  /**
+   * Check if partition can be placed on the instance.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @return boolean - True if the partition can be placed, False otherwise
+   */
+  public boolean isInstanceCapacityAvailable(String instanceName, Map<String, Integer> partitionCapacity);
+
+  /**
+   * Reduce the available capacity by specified Partition Capacity Map.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @returns boolean - True if successfully updated partition capacity, false otherwise.
+   */
+  public boolean reduceAvailableInstanceCapacity(String instanceName, Map<String, Integer> partitionCapacity);

Review Comment:
   I like the suggestion of having a "manager" class. But that pattern is not followed by all the existing DataProviders. Current DataProviders are not pure interface and are all stateful too. I am sure, you must have seen: https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/dataproviders/ResourceControllerDataProvider.java
   
   Let me know if the above makes sense. One is code organization and another is method scopes and purpose. Please do review the existing data provider and let me know. thanks,
   komal



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199158050


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }

Review Comment:
   changed the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on PR #2493:
URL: https://github.com/apache/helix/pull/2493#issuecomment-1559238810

   @qqu0127 - i have resolved the comments. can i please get your review? thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1198270094


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }

Review Comment:
   how about `partitionCapacity.getOrDefault(key, 0)`? 



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+      }
+    }
+    return true;
+  }
+
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    boolean modified = false;
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+        if (partCapacity != 0) {
+          instanceCapacity.put(key, instanceCapacity.get(key) - partCapacity);
+          modified = true;
+        }
+      }
+    }
+    if (modified) {
+      _perInstance.put(instance, instanceCapacity);
+    }

Review Comment:
   We may still have partial update with this. `instanceCapacity` is just a reference, not new copy



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {

Review Comment:
   Do we skip or hard fail if partition contains additional weight key than specified in instance?



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+      }
+    }
+    return true;
+  }
+
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {

Review Comment:
   nit: tag `@Override` for this and above.



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+      }
+    }
+    return true;
+  }
+
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    boolean modified = false;
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);

Review Comment:
   nit: let's use unboxed type `int` directly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu merged pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2493:
URL: https://github.com/apache/helix/pull/2493


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1206834163


##########
helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java:
##########
@@ -460,4 +460,9 @@ public Map<String, ResourceAssignment> getAssignment(Set<String> resourceSet) {
 
     return currentStateAssignment;
   }
+
+
+  public Map<String, Map<Partition, Map<String, Message>>> getPendingMessages() {

Review Comment:
   Thanks for pointing that out. I wanted to get all the pending messages to process in one go. The flow is: You have capacity - <all pending messages> - this is how we start  once the assignment has been calculated.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199157489


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {

Review Comment:
   will do this as part of change-2.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199114338


##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/InstanceCapacityDataProvider.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.helix.controller.dataproviders;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+
+/**
+ * An interface to provide capacity data for instance.
+ * It will consider the pending state transition and assigned partitions capacity.
+ * It will be dynamic and will always provide the available "headroom".
+ *
+ * The actual implementation will be stateful.
+ */
+public interface InstanceCapacityDataProvider {
+
+ /**
+   * Get the instance remaining capacity. 
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName);
+
+  /**
+   * Check if partition can be placed on the instance.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @return boolean - True if the partition can be placed, False otherwise
+   */
+  public boolean isInstanceCapacityAvailable(String instanceName, Map<String, Integer> partitionCapacity);
+
+  /**
+   * Reduce the available capacity by specified Partition Capacity Map.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @returns boolean - True if successfully updated partition capacity, false otherwise.
+   */
+  public boolean reduceAvailableInstanceCapacity(String instanceName, Map<String, Integer> partitionCapacity);

Review Comment:
   @qqu0127 - can we resolve this? I can re-structure the code as part of 2.0 effort.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199116399


##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/InstanceCapacityDataProvider.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.helix.controller.dataproviders;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+
+/**
+ * An interface to provide capacity data for instance.
+ * It will consider the pending state transition and assigned partitions capacity.
+ * It will be dynamic and will always provide the available "headroom".
+ *
+ * The actual implementation will be stateful.
+ */
+public interface InstanceCapacityDataProvider {
+
+ /**
+   * Get the instance remaining capacity. 
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName);
+
+  /**
+   * Check if partition can be placed on the instance.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @return boolean - True if the partition can be placed, False otherwise
+   */
+  public boolean isInstanceCapacityAvailable(String instanceName, Map<String, Integer> partitionCapacity);
+
+  /**
+   * Reduce the available capacity by specified Partition Capacity Map.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @returns boolean - True if successfully updated partition capacity, false otherwise.
+   */
+  public boolean reduceAvailableInstanceCapacity(String instanceName, Map<String, Integer> partitionCapacity);

Review Comment:
   yes, feel free to resolve, this is not blocking. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199117366


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {

Review Comment:
   Capacity provider which always gives you availableCapacity.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199163747


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {

Review Comment:
   This seems a little weird from https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/util/WagedValidationUtil.java#L85. where we check required keys and throw exception.
   I don't think it's a blocking issue though, the check is done regardless. Feel free to resolve.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] junkaixue commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199206342


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;

Review Comment:
   shall we name it _instanceCapacityMap?



##########
helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java:
##########
@@ -460,4 +460,9 @@ public Map<String, ResourceAssignment> getAssignment(Set<String> resourceSet) {
 
     return currentStateAssignment;
   }
+
+
+  public Map<String, Map<Partition, Map<String, Message>>> getPendingMessages() {
+    return _pendingMessageMap;

Review Comment:
   Return Collections.umodifiableMap(_pendingMessageMap);



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);

Review Comment:
   Capacity set to 0 sounds a little bit awkward. My understanding is if non-waged compute, they can skip this capacity check.



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  @Override
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  @Override
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      Integer partCapacity = partitionCapacity.getOrDefault(key, 0);
+      if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+        return false;
+      }
+    }
+    return true;
+  }
+
+  @Override
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.getOrDefault(key, 0);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+        if (partCapacity != 0) {
+          instanceCapacity.put(key, instanceCapacity.get(key) - partCapacity);
+        }

Review Comment:
   This can be combined into short one:
   
   if (partCapacity != 0) {
       if (instanceCapacity.get(key) < partCapacity) {
            return false;
       }
       instanceCapacity.put(key, instanceCapacity.get(key) - partCapacity);
   }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1200668783


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);

Review Comment:
   Please refer to https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/util/DelayedRebalanceUtil.java#L440 to merge resource config and config in ideal state. 



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }

Review Comment:
   Are the checks left for part 2?



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  @Override
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  @Override
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      Integer partCapacity = partitionCapacity.getOrDefault(key, 0);

Review Comment:
   Integer -> int



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;

Review Comment:
   these can be final?



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  @Override
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  @Override
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      Integer partCapacity = partitionCapacity.getOrDefault(key, 0);
+      if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+        return false;
+      }
+    }
+    return true;
+  }
+
+  @Override
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.getOrDefault(key, 0);

Review Comment:
   Integer -> int



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1200774465


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  private Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  @Override
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  @Override
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      Integer partCapacity = partitionCapacity.getOrDefault(key, 0);
+      if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+        return false;
+      }
+    }
+    return true;
+  }
+
+  @Override
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.getOrDefault(key, 0);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+        if (partCapacity != 0) {
+          instanceCapacity.put(key, instanceCapacity.get(key) - partCapacity);
+        }

Review Comment:
   addressed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] qqu0127 commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1198264817


##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/InstanceCapacityDataProvider.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.helix.controller.dataproviders;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+
+/**
+ * An interface to provide capacity data for instance.
+ * It will consider the pending state transition and assigned partitions capacity.
+ * It will be dynamic and will always provide the available "headroom".
+ *
+ * The actual implementation will be stateful.
+ */
+public interface InstanceCapacityDataProvider {
+
+ /**
+   * Get the instance remaining capacity. 
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName);
+
+  /**
+   * Check if partition can be placed on the instance.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @return boolean - True if the partition can be placed, False otherwise
+   */
+  public boolean isInstanceCapacityAvailable(String instanceName, Map<String, Integer> partitionCapacity);
+
+  /**
+   * Reduce the available capacity by specified Partition Capacity Map.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @returns boolean - True if successfully updated partition capacity, false otherwise.
+   */
+  public boolean reduceAvailableInstanceCapacity(String instanceName, Map<String, Integer> partitionCapacity);

Review Comment:
   I agree, all current data providers follow that pattern. It's just something to keep in mind that, if we have some other weights source provider, e.g. the next gen rebalancer, we need to again build the data provider + stateful tracking too. 
   With that said, the current structure also looks good. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1199185038


##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {
+    Map<String, Map<Partition, Map<String, Message>>> pendingMsgs = currentState.getPendingMessages();
+    
+    for (String resource : pendingMsgs.keySet()) {
+      Map<Partition, Map<String, Message>> partitionMsgs = pendingMsgs.get(resource);
+
+      for (Partition partition : partitionMsgs.keySet()) {
+        String partitionName = partition.getPartitionName();
+
+        // Get Partition Weight
+        Map<String, Integer> partCapacity = getPartitionCapacity(resource, partitionName);
+
+        // TODO - check
+        Map<String, Message> msgs = partitionMsgs.get(partition);
+        // TODO - Check
+        for (String instance : msgs.keySet()) {
+           reduceAvailableInstanceCapacity(instance, partCapacity);
+        }
+      }
+    }
+  }
+
+  /**
+   * Get the partition capacity given Resource and Partition name.
+   */
+  private Map<String, Integer> getPartitionCapacity(String resource, String partition) {
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    ResourceConfig resourceConfig = _cache.getResourceConfig(resource);
+
+
+    // Parse the entire capacityMap from ResourceConfig
+    Map<String, Map<String, Integer>> capacityMap;
+    try {
+      capacityMap = resourceConfig.getPartitionCapacityMap();
+    } catch (IOException ex) {
+      return createDefaultParticipantWeight();
+    }
+    return WagedValidationUtil.validateAndGetPartitionCapacity(partition, resourceConfig, capacityMap, clusterConfig);
+  }
+
+  /**
+   * Get the instance remaining capacity.
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName) {
+    return _perInstance.get(instanceName);
+  }
+
+  public boolean isInstanceCapacityAvailable(String instance, Map<String, Integer> partitionCapacity) {
+    Map<String, Integer> instanceCapacity = _perInstance.get(instance);
+    for (String key : instanceCapacity.keySet()) {
+      if (partitionCapacity.containsKey(key)) {
+        Integer partCapacity = partitionCapacity.get(key);
+        if (partCapacity != 0 && instanceCapacity.get(key) < partCapacity) {
+          return false;
+        }
+      }
+    }
+    return true;
+  }
+
+  public boolean reduceAvailableInstanceCapacity(String instance, Map<String, Integer> partitionCapacity) {

Review Comment:
   thanks.. i had forgotten to add in last commit. re-did it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] rahulrane50 commented on a diff in pull request #2493: Waged - Hard constraint - Fix for n - n+1 issue.

Posted by "rahulrane50 (via GitHub)" <gi...@apache.org>.
rahulrane50 commented on code in PR #2493:
URL: https://github.com/apache/helix/pull/2493#discussion_r1204883835


##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/InstanceCapacityDataProvider.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.helix.controller.dataproviders;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.Map;
+
+/**
+ * An interface to provide capacity data for instance.
+ * It will consider the pending state transition and assigned partitions capacity.
+ * It will be dynamic and will always provide the available "headroom".
+ *
+ * The actual implementation will be stateful.
+ */
+public interface InstanceCapacityDataProvider {
+
+ /**
+   * Get the instance remaining capacity. 
+   * Capacity and weight both are represented as Key-Value.
+   * Returns the capacity map of available head room for the instance.
+   * @param instanceName - instance name to query
+   * @return Map<String, Integer> - capacity pair for all defined attributes for the instance.
+   */
+  public Map<String, Integer> getInstanceAvailableCapacity(String instanceName);
+
+  /**
+   * Check if partition can be placed on the instance.
+   *
+   * @param instanceName - instance name 
+   * @param partitionCapacity - Partition capacity expresed in capacity map.
+   * @return boolean - True if the partition can be placed, False otherwise
+   */
+  public boolean isInstanceCapacityAvailable(String instanceName, Map<String, Integer> partitionCapacity);

Review Comment:
   nit. Should it be "partitionWeight" but "partitionCapacity"? I see this class as one which provides "available" capacity of an instance and this method as to check given partition "weight" would it ok to place it on that instance.



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private Map<String, Map<String, Integer>> _perInstance;
+  private ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {
+    _cache = clusterData;
+    _perInstance = new HashMap<>();
+
+    ClusterConfig clusterConfig = _cache.getClusterConfig();
+    for (InstanceConfig instanceConfig : _cache.getInstanceConfigMap().values()) {
+      Map<String, Integer> instanceCapacity =
+        WagedValidationUtil.validateAndGetInstanceCapacity(clusterConfig, instanceConfig);
+      _perInstance.put(instanceConfig.getInstanceName(), instanceCapacity);
+    }
+  }
+
+  /**
+   * Create Default Capacity Map.
+   * This is a utility method to create a default capacity map matching instance capacity map for participants.
+   * This is required as non-WAGED partitions will be placed on same instance and we don't know their actual capacity.
+   * This will generate default values of 0 for all the capacity keys.
+   */
+  public Map<String, Integer> createDefaultParticipantWeight() {
+    // copy the value of first Instance capacity.
+    Map<String, Integer> partCapacity = new HashMap<>(_perInstance.values().iterator().next());
+
+    // reset the value of all capacity to 0.
+    for (String key : partCapacity.keySet()) {
+      partCapacity.put(key, 0);
+    }
+    return partCapacity;
+  }
+
+  /**
+   * Process the pending messages based on the Current states
+   * @param currentState - Current state of the resources.
+   */
+  public void processPendingMessages(CurrentStateOutput currentState) {

Review Comment:
   +1 to this.
   Everything else makes sense to me but this method looks like one method which goes over "all" resources and then reduce available instance capacity. 
   First we can update method name that suggests so. (may be like iteratePendingMsgsAndUpdateInstanceCapacity so something like that). And if that's too long name then atleast description should be updated. 



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,164 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {

Review Comment:
   If I understand correctly then sole purpose of this class is update available instance capacity based on pending messages. If that's true then we can change class name which closely presents that or else add comments for that. 



##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/WagedInstanceCapacity.java:
##########
@@ -0,0 +1,160 @@
+package org.apache.helix.controller.rebalancer.waged;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.apache.helix.controller.stages.CurrentStateOutput;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.Message;
+import org.apache.helix.model.Partition;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.controller.dataproviders.InstanceCapacityDataProvider;
+import org.apache.helix.controller.dataproviders.ResourceControllerDataProvider;
+import org.apache.helix.controller.rebalancer.util.WagedValidationUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class WagedInstanceCapacity implements InstanceCapacityDataProvider {
+  private static final Logger LOG = LoggerFactory.getLogger(WagedInstanceCapacity.class);
+
+  // Available Capacity per Instance
+  private final Map<String, Map<String, Integer>> _instanceCapacityMap;
+  private final ResourceControllerDataProvider _cache;
+
+  public WagedInstanceCapacity(ResourceControllerDataProvider clusterData) {

Review Comment:
   Another dumb que i had was AssignableNode also maintains this right? I understand it doesn't represent "temporary state" of an node but "ideally" we should update "remaining capacity" of the instance only when actual replica is mapped to that instance i..,e successful ST. here we make assumption that pending ST is going to be successful and hence proactively reduce instance capacity which may or may not be true (because that's actually going to be decided in your part 2 solution DelayedAutoRebalancer PR). I can suggest to clearly update the comments that it's temp "available" capacity of the instance.



##########
helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java:
##########
@@ -460,4 +460,9 @@ public Map<String, ResourceAssignment> getAssignment(Set<String> resourceSet) {
 
     return currentStateAssignment;
   }
+
+
+  public Map<String, Map<Partition, Map<String, Message>>> getPendingMessages() {

Review Comment:
   We do have getPendingMessageMap() method in CurrentStateOutput.java right? Do we want another getter which gets for all resources?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org