You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/02/21 23:29:26 UTC
[GitHub] [helix] NealSun96 opened a new pull request #1652: Add TopStateUsage constraint to Waged
NealSun96 opened a new pull request #1652:
URL: https://github.com/apache/helix/pull/1652
### Issues
- [x] My PR addresses the following Helix issues and references them in the PR description:
fix #1651
### Description
- [x] Here are some details about my PR, including screenshots of any UI changes:
In current Waged implementation, TopStateAntiAffinityConstraint is in charge of spreading out top state replicas and it's based on replica count instead of weights. Originally, this decision was made because we assume replica weight evenness is already handled by MaxCapacityUsageInstanceConstraint; however, real usage has proven that top state replica weight evenness is very bad, while regular state replica weight evenness is good.
Therefore, we need a weight-based constraint to spread top state replicas out.
### Tests
- The following is the result of the "mvn test" command on the appropriate module:
(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang merged pull request #1652: Add TopStateUsage constraint to Waged
Posted by GitBox <gi...@apache.org>.
jiajunwang merged pull request #1652:
URL: https://github.com/apache/helix/pull/1652
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1652: Add TopStateUsage constraint to Waged
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1652:
URL: https://github.com/apache/helix/pull/1652#discussion_r582362522
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/ConstraintBasedAlgorithmFactory.java
##########
@@ -41,8 +41,9 @@
put(PartitionMovementConstraint.class.getSimpleName(), 2f);
put(InstancePartitionsCountConstraint.class.getSimpleName(), 1f);
put(ResourcePartitionAntiAffinityConstraint.class.getSimpleName(), 1f);
- put(ResourceTopStateAntiAffinityConstraint.class.getSimpleName(), 3f);
- put(MaxCapacityUsageInstanceConstraint.class.getSimpleName(), 5f);
+ put(ResourceTopStateAntiAffinityConstraint.class.getSimpleName(), 0f);
Review comment:
Let's just remove the ResourceTopStateAntiAffinityConstraint from this list.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/model/AssignableNode.java
##########
@@ -226,13 +233,16 @@ public int getAssignedReplicaCount() {
* For example, if the current node usage is {CPU: 0.9, MEM: 0.4, DISK: 0.6}. Then this call shall
* return 0.9.
* @param newUsage the proposed new additional capacity usage.
+ * @param isTopState whether the returned utilization is for top state partitions only
* @return The highest utilization number of the node among all the capacity category.
*/
- public float getProjectedHighestUtilization(Map<String, Integer> newUsage) {
+ public float getProjectedHighestUtilization(Map<String, Integer> newUsage, boolean isTopState) {
Review comment:
Using boolean input to switch branch is not a clear design overall.
Please add another method for getProjectedHighestTopStateUtilization(). To minimize duplicate code, we can have a private method that contains the common part. We can put Map<String, Integer> remainingCapacity as an input parameter for the private method, so that the later logic will be the same for both public methods.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/model/AssignableNode.java
##########
@@ -311,13 +321,17 @@ private void addToAssignmentRecord(AssignableReplica replica) {
}
}
- private void updateRemainingCapacity(String capacityKey, int usage) {
- if (!_remainingCapacity.containsKey(capacityKey)) {
- //if the capacityKey belongs to replicas does not exist in the instance's capacity,
- // it will be treated as if it has unlimited capacity of that capacityKey
- return;
+ private void updateRemainingCapacity(Map<String, Integer> usedCapacity, boolean isRelease,
Review comment:
Same here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on pull request #1652: Add TopStateUsage constraint to Waged
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on pull request #1652:
URL: https://github.com/apache/helix/pull/1652#issuecomment-788121403
This PR is ready to be merged, approved by @jiajunwang
Final commit message:
## Add top state usage constraint to Waged ##
Add new top state weight based constraint to Waged to ensure top state weight evenness.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1652: Add TopStateUsage constraint to Waged
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1652:
URL: https://github.com/apache/helix/pull/1652#discussion_r583164602
##########
File path: helix-core/src/test/java/org/apache/helix/integration/rebalancer/WagedRebalancer/TestWagedRebalance.java
##########
@@ -135,6 +135,16 @@ protected void addInstanceConfig(String storageNodeName, int seqNo, int tagCount
@Test
public void test() throws Exception {
+ HelixDataAccessor dataAccessor = new ZKHelixDataAccessor(CLUSTER_NAME, _baseAccessor);
Review comment:
Put this logic into the beforeClass() method as part of the initial configuration process? Or you shall remove this config once the test is done. Otherwise, the configuration might impact the other test cases unexpectedly.
##########
File path: helix-core/src/test/java/org/apache/helix/controller/rebalancer/waged/model/TestClusterContext.java
##########
@@ -68,6 +68,8 @@ public void testNormalUsage() throws IOException {
.addPartitionToFaultZone(_testFaultZoneId, replica.getResourceName(),
replica.getPartitionName()));
Assert.assertEquals(context.getAssignmentForFaultZoneMap(), expectedFaultZoneMap);
+ Assert.assertEquals(context.getEstimatedMaxUtilization(), 16.0 / 20.0, 0.005);
Review comment:
nit, please roughly explain the numbers here. They are not necessarily clear to the other readers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org