You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/05/20 23:49:50 UTC
[GitHub] [helix] junkaixue opened a new pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
junkaixue opened a new pull request #1750:
URL: https://github.com/apache/helix/pull/1750
### Issues
- [X] My PR addresses the following Helix issues and references them in the PR description:
resolves #343
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
Make Pipeline in a correct order / minor improvements and fixes tests
1. Make pipeline running in a correct orde to process computation
2. Add "DROPPED" case in the mapping to reflect the real case.
3. Make unfound state from current state mapping to INITIAL state.
3. Make partition sorting by name as the last rule.
3. Fixes test cases.
### Tests
- [ ] The following tests are written for this issue:
- The following is the result of the "mvn test" command on the appropriate module:
TBD
### Changes that Break Backward Compatibility (Optional)
- My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1750:
URL: https://github.com/apache/helix/pull/1750#discussion_r638358760
##########
File path: helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java
##########
@@ -513,12 +513,14 @@ private static PipelineRegistry createDefaultRegistry(String pipelineName) {
// rebalance pipeline
Pipeline rebalancePipeline = new Pipeline(pipelineName);
rebalancePipeline.addStage(new BestPossibleStateCalcStage());
- rebalancePipeline.addStage(new IntermediateStateCalcStage());
// Need to add MaintenanceRecoveryStage here because MAX_PARTITIONS_PER_INSTANCE check could
// only occur after IntermediateStateCalcStage calculation
rebalancePipeline.addStage(new MaintenanceRecoveryStage());
rebalancePipeline.addStage(new ResourceMessageGenerationPhase());
rebalancePipeline.addStage(new MessageSelectionStage());
+ // Move the IntermediateStateCalcStage after message selection
Review comment:
nit, future code readers won't see this PR directly. Let's change the description to be "The IntermediateStateCalcStage should be after message selection."
##########
File path: helix-core/src/test/java/org/apache/helix/controller/stages/TestRecoveryLoadBalance.java
##########
@@ -57,7 +57,10 @@
private final String STATE_MODEL = "statemodel";
private ClusterConfig _clusterConfig;
- @Test(dataProvider = "recoveryLoadBalanceInput")
+ // Disable the test since recovery rebalance is not able to block load rebalance.
+ // TODO: if we decide to support that by replica level, we will have a different config for that and this test can be
Review comment:
This is the pain point that we want to address with this change. I suggest change the test to validate the reverse way. So you don't have to disable the test, and the new change will be protected by more coverage.
##########
File path: helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ResourceMonitor.java
##########
@@ -422,22 +440,42 @@ public long getMissingReplicaPartitionGauge() {
return _numLessReplicaPartitions.getValue();
}
+ @Deprecated
Review comment:
Given the assumption that user will always rely on the MBean Server, I think we can directly remove these methods? They are now only for testing.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/stages/IntermediateStateCalcStage.java
##########
@@ -889,17 +893,23 @@ private int getIdealStateMatched(Partition partition) {
private void computeIntermediateMap(PartitionStateMap intermediateStateMap,
Review comment:
To avoid duplicate code, how about generalizing computeIntermediateMap so it takes only one map and always computes with the same logic?
for (Map.Entry<Partition, Map<String, Message>> entry : inputMessageMap.entrySet()) {
entry.getValue().entrySet().stream().forEach(e -> {
if (!e.getValue().getToState().equals(HelixDefinedState.DROPPED.name())) {
intermediateStateMap.setState(entry.getKey(), e.getValue().getTgtName(), e.getValue().getToState());
} else {
intermediateStateMap.getStateMap().get(entry.getKey()).remove(e.getValue().getTgtName());
}
});
}
##########
File path: helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ResourceMonitor.java
##########
@@ -370,10 +388,10 @@ public void updateStateHandoffStats(MonitorState monitorState, long totalDuratio
public void updateRebalancerStats(long numPendingRecoveryRebalancePartitions,
long numPendingLoadRebalancePartitions, long numRecoveryRebalanceThrottledPartitions,
long numLoadRebalanceThrottledPartitions) {
- _numPendingRecoveryRebalancePartitions.updateValue(numPendingRecoveryRebalancePartitions);
- _numPendingLoadRebalancePartitions.updateValue(numPendingLoadRebalancePartitions);
- _numRecoveryRebalanceThrottledPartitions.updateValue(numRecoveryRebalanceThrottledPartitions);
- _numLoadRebalanceThrottledPartitions.updateValue(numLoadRebalanceThrottledPartitions);
+ _numPendingRecoveryRebalanceReplicas.updateValue(numPendingRecoveryRebalancePartitions);
Review comment:
Why leave all the deprecated fields? They will be zero forever, right?
I prefer removing them. Or if strict backward compatibility is to be ensured, then just keep updating their values.
##########
File path: helix-core/src/test/java/org/apache/helix/controller/stages/TestIntermediateStateCalcStage.java
##########
@@ -25,12 +25,14 @@
import java.util.Map;
import com.google.common.collect.ImmutableList;
+import java.util.UUID;
Review comment:
nit, not used?
##########
File path: helix-core/src/main/java/org/apache/helix/controller/stages/IntermediateStateCalcStage.java
##########
@@ -467,7 +467,8 @@ private void chargePendingTransition(Resource resource, CurrentStateOutput curre
for (Message message : pendingMessages) {
StateTransitionThrottleConfig.RebalanceType rebalanceType =
getRebalanceTypePerMessage(requiredStates, message, currentStateMap);
- String currentState = currentStateMap.get(message.getTgtName());
+ String currentState = currentStateMap.get(message.getTgtName()) == null ? stateModelDefinition.getInitialState()
+ : currentStateMap.get(message.getTgtName());
Review comment:
nit, how about just add a null check like this, I think it's simpler and sightly more efficient.
String currentState = currentStateMap.get(message.getTgtName());
if (currentState == null) {
currentState = stateModelDefinition.getInitialState();
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a change in pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1750:
URL: https://github.com/apache/helix/pull/1750#discussion_r638378537
##########
File path: helix-core/src/test/java/org/apache/helix/controller/stages/TestRecoveryLoadBalance.java
##########
@@ -57,7 +57,10 @@
private final String STATE_MODEL = "statemodel";
private ClusterConfig _clusterConfig;
- @Test(dataProvider = "recoveryLoadBalanceInput")
+ // Disable the test since recovery rebalance is not able to block load rebalance.
+ // TODO: if we decide to support that by replica level, we will have a different config for that and this test can be
Review comment:
I will have a new test class in a different PR already covered those recovery cases.
Disable the test is keep the possibility of supporting the feature again. As I said, this feature is not determined to implement or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a change in pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1750:
URL: https://github.com/apache/helix/pull/1750#discussion_r638377677
##########
File path: helix-core/src/main/java/org/apache/helix/controller/stages/IntermediateStateCalcStage.java
##########
@@ -889,17 +893,23 @@ private int getIdealStateMatched(Partition partition) {
private void computeIntermediateMap(PartitionStateMap intermediateStateMap,
Review comment:
@jiajunwang They are different structures of the input. One is Map<String, Message> and another is Map<String, String>.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a change in pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1750:
URL: https://github.com/apache/helix/pull/1750#discussion_r638378537
##########
File path: helix-core/src/test/java/org/apache/helix/controller/stages/TestRecoveryLoadBalance.java
##########
@@ -57,7 +57,10 @@
private final String STATE_MODEL = "statemodel";
private ClusterConfig _clusterConfig;
- @Test(dataProvider = "recoveryLoadBalanceInput")
+ // Disable the test since recovery rebalance is not able to block load rebalance.
+ // TODO: if we decide to support that by replica level, we will have a different config for that and this test can be
Review comment:
I will have a new test class already covered most of cases.
Disable the test is keep the possibility of supporting the feature again. As I said, this feature is not determined to implement or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue merged pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
junkaixue merged pull request #1750:
URL: https://github.com/apache/helix/pull/1750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a change in pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1750:
URL: https://github.com/apache/helix/pull/1750#discussion_r638373647
##########
File path: helix-core/src/main/java/org/apache/helix/controller/stages/IntermediateStateCalcStage.java
##########
@@ -889,17 +893,23 @@ private int getIdealStateMatched(Partition partition) {
private void computeIntermediateMap(PartitionStateMap intermediateStateMap,
Review comment:
@jiajunwang They are different structure of the code. They are different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1750:
URL: https://github.com/apache/helix/pull/1750#discussion_r638395639
##########
File path: helix-core/src/test/java/org/apache/helix/controller/stages/TestRecoveryLoadBalance.java
##########
@@ -57,7 +57,10 @@
private final String STATE_MODEL = "statemodel";
private ClusterConfig _clusterConfig;
- @Test(dataProvider = "recoveryLoadBalanceInput")
+ // Disable the test since recovery rebalance is not able to block load rebalance.
+ // TODO: if we decide to support that by replica level, we will have a different config for that and this test can be
Review comment:
I would suggest removing this test given we already have test coverage. First, there is no plan or discussion about re-enabling it. Secondly, I think simply block loadbalance because of recovery without considering throttling is not correct behavior. If we are going to do a similar restriction, we would do it through throttling anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue commented on a change in pull request #1750: [Replica Level Throttle] Make Pipeline in a correct order and fixes tests
Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1750:
URL: https://github.com/apache/helix/pull/1750#discussion_r638378537
##########
File path: helix-core/src/test/java/org/apache/helix/controller/stages/TestRecoveryLoadBalance.java
##########
@@ -57,7 +57,10 @@
private final String STATE_MODEL = "statemodel";
private ClusterConfig _clusterConfig;
- @Test(dataProvider = "recoveryLoadBalanceInput")
+ // Disable the test since recovery rebalance is not able to block load rebalance.
+ // TODO: if we decide to support that by replica level, we will have a different config for that and this test can be
Review comment:
In this PR, I would like to stablize the existing tests. I will have a new test class in a different PR already covered those recovery cases.
Disable the test is keep the possibility of supporting the feature again. As I said, this feature is not determined to implement or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org