You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "GrantPSpencer (via GitHub)" <gi...@apache.org> on 2024/07/24 19:26:25 UTC
[PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
GrantPSpencer opened a new pull request, #2848:
URL: https://github.com/apache/helix/pull/2848
### Issues
- [ ] My PR addresses the following Helix issues and references them in the PR description:
With the recent changes to InstanceOperation, Helix allows users to set their instances to DISABLE where all partitions are gracefully transited to the offline state. Likewise, customers can also disable specific partitions to transit them to offline (CRUSHED) or move them off the node (WAGED). However, because there can only be 1 active InstanceOperation on a node at a time, a user cannot DISABLE an evacuating or SWAP_IN instance without overriding that operation. There are scenarios where a user may want to ensure that the instance being operated on immediately downward state transits its partitions or does not receive any upward state transitions while it is being operated on.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI changes:
This change adds support for the ALL_RESOURCES key in the disabled partitions map of an instance's config. This change mirrors the current behavior of disabling a specific partition on a node, but extends that to all partitions in the cluster.
The main area this change effects are:
WAGED placement calculation - AssignableNode and ReplicaActivateConstraint
DelayedAutoRebalancer - computeBestPossiblePartitionState (responsible for forcing partitions down to offline)
There will likely need to be discussion had on how to best reduce complexity of adding the "ALL_RESOURCES" key. Future code may need to explicitly check for this key depending on what methods they are leveraging. which will likely not be obvious for developers
### Tests
- [ ] The following tests are written for this issue:
**Basic Functionality**
testDisableAllPartitions in TestDisablePartitions.java
**Test Asserting Behavior Alongside Instance Operation**
testEvacuateWithDisabledPartition in TestInstanceOperation.java
### Changes that Break Backward Compatibility (Optional)
- My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
N/A
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on PR #2848:
URL: https://github.com/apache/helix/pull/2848#issuecomment-2251251560
Pull request approved @junkaixue, @zpinto
Commit message: Add support for ALL_RESOURCES key to disabled partitions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on code in PR #2848:
URL: https://github.com/apache/helix/pull/2848#discussion_r1690426652
##########
helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestInstanceOperation.java:
##########
@@ -1361,6 +1361,96 @@ public void testEvacuationWithOfflineInstancesInCluster() throws Exception {
dropTestDBs(ImmutableSet.of("TEST_DB3_DELAYED_CRUSHED", "TEST_DB4_DELAYED_WAGED"));
}
+ @Test(dependsOnMethods = "testEvacuationWithOfflineInstancesInCluster")
+ public void testEvacuateWithDisabledPartition() throws Exception {
+ System.out.println(
+ "START TestInstanceOperation.testEvacuateWithDisabledPartition() at " + new Date(
+ System.currentTimeMillis()));
+ StateTransitionCountStateModelFactory stateTransitionCountStateModelFactory = new StateTransitionCountStateModelFactory();
+ removeOfflineOrInactiveInstances();
+ String testCrushedDBName = "testEvacuateWithDisabledPartition_CRUSHED_DB0";
+ String testWagedDBName = "testEvacuateWithDisabledPartition_WAGED_DB1";
+ String toDisableThenEvacuateInstanceName = "disable_then_evacuate_host";
+ addParticipant(toDisableThenEvacuateInstanceName, stateTransitionCountStateModelFactory);
+ MockParticipantManager toDisableThenEvacuateParticipant = _participants.get(_participants.size() - 1);
+
+ List<String> testResources = Arrays.asList(testCrushedDBName, testWagedDBName);
+ createResourceWithDelayedRebalance(CLUSTER_NAME, testCrushedDBName, "MasterSlave",
+ PARTITIONS, REPLICA, REPLICA-1, 200000, CrushEdRebalanceStrategy.class.getName());
+ createResourceWithWagedRebalance(CLUSTER_NAME, testWagedDBName, "MasterSlave", PARTITIONS,
+ REPLICA, REPLICA-1);
+
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+ int upwardSTCountBeforeDisableThenEvacuate = stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter();
+ int downwardSTCountBeforeDisableThenEvacuate = stateTransitionCountStateModelFactory.getDownwardStateTransitionCounter();
+
+
+ InstanceConfig instanceConfig = _gSetupTool.getClusterManagementTool().getInstanceConfig(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName);
+ instanceConfig.setInstanceEnabledForPartition(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY, "", false);
+ _gSetupTool.getClusterManagementTool().setInstanceConfig(CLUSTER_NAME, toDisableThenEvacuateInstanceName, instanceConfig);
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+ // EV should not have disabled instance above the lowest state (OFFLINE)
+ verifier(() -> {
+ for (String resource : testResources) {
+ ExternalView ev = _gSetupTool.getClusterManagementTool().getResourceExternalView(CLUSTER_NAME, resource);
+ for (String partition : ev.getPartitionSet()) {
+ if (ev.getStateMap(partition).containsKey(toDisableThenEvacuateInstanceName) && !ev.getStateMap(partition).
+ get(toDisableThenEvacuateInstanceName).equals("OFFLINE")) {
+ return false;
+ }
+ }
+ }
+ return true;
+ }, 5000);
+
+ // Assert node received downward state transitions and no upward transitions
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
+ upwardSTCountBeforeDisableThenEvacuate, "Upward state transitions should not have been received");
+ Assert.assertTrue(stateTransitionCountStateModelFactory.getDownwardStateTransitionCounter() >
+ downwardSTCountBeforeDisableThenEvacuate, "Should have received downward state transitions");
+
+ _gSetupTool.getClusterManagementTool().setInstanceOperation(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName, InstanceConstants.InstanceOperation.EVACUATE);
+
+ verifier(() -> _admin.isEvacuateFinished(CLUSTER_NAME, toDisableThenEvacuateInstanceName), 30000);
+
+ // Assert node received no upward state transitions after evacuation was called on already disabled node
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
+ upwardSTCountBeforeDisableThenEvacuate, "Upward state transitions should not have been received");
+
+ // Re-enable all partitions for the instance
+ instanceConfig = _gSetupTool.getClusterManagementTool().getInstanceConfig(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName);
+ instanceConfig.setInstanceEnabledForPartition(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY, "", true);
+ _gSetupTool.getClusterManagementTool().setInstanceConfig(CLUSTER_NAME, toDisableThenEvacuateInstanceName, instanceConfig);
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+
+ // Assert node received no upward state transitions after re-enabled partitions
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
+ upwardSTCountBeforeDisableThenEvacuate, "Upward state transitions should not have been received");
+
+ // Disable all partitions for the instance again
+ instanceConfig = _gSetupTool.getClusterManagementTool().getInstanceConfig(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName);
+ instanceConfig.setInstanceEnabledForPartition(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY, "", true);
+ _gSetupTool.getClusterManagementTool().setInstanceConfig(CLUSTER_NAME, toDisableThenEvacuateInstanceName, instanceConfig);
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+
+ // Assert node received no upward state transitions after disabling already evacuated node
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
Review Comment:
We should also assert no downward since all the partitions should already be dropped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2848:
URL: https://github.com/apache/helix/pull/2848#discussion_r1690498610
##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/BaseControllerDataProvider.java:
##########
@@ -1078,12 +1081,16 @@ private void updateDisabledInstances(Collection<InstanceConfig> allInstanceConfi
// Move the calculating disabled instances to refresh
_disabledInstanceForPartitionMap.clear();
_disabledInstanceSet.clear();
+ _disabledInstancesForAllPartitionsSet.clear();
for (InstanceConfig config : allInstanceConfigs) {
Map<String, List<String>> disabledPartitionMap = config.getDisabledPartitionsMap();
- if (config.getInstanceOperation().getOperation()
- .equals(InstanceConstants.InstanceOperation.DISABLE)) {
+ // Treat instance as disabled if it has "DISABLE" operation or "ALL_RESOURCES" in the disabled partition map
+ if (config.getInstanceOperation().getOperation().equals(InstanceConstants.InstanceOperation.DISABLE)) {
_disabledInstanceSet.add(config.getInstanceName());
}
+ if (disabledPartitionMap.containsKey(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY)) {
+ _disabledInstancesForAllPartitionsSet.add(config.getInstanceName());
Review Comment:
Not make it in base controller. This resource assignment, make it in resource controller provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue merged PR #2848:
URL: https://github.com/apache/helix/pull/2848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on PR #2848:
URL: https://github.com/apache/helix/pull/2848#issuecomment-2250585664
> @GrantPSpencer This test failed: Test failed: testDisablePartition(org.apache.helix.integration.TestDisablePartition) Time elapsed: 30.279 s <<< FAILURE!
>
> It is related to your change. Make sure this is fixed.
I was not cleaning up the resources I added solely for the new test method (testDisableAllPartitions) in testDisablePartition class. This caused a downstream issue seen in the failed test. Pushing change now and will trigger a few cI runs on my personal fork to confirm not flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on PR #2848:
URL: https://github.com/apache/helix/pull/2848#issuecomment-2249428741
@GrantPSpencer This test failed: Test failed: testDisablePartition(org.apache.helix.integration.TestDisablePartition) Time elapsed: 30.279 s <<< FAILURE!
It is related to your change. Make sure this is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2848:
URL: https://github.com/apache/helix/pull/2848#discussion_r1690603790
##########
helix-core/src/main/java/org/apache/helix/controller/dataproviders/BaseControllerDataProvider.java:
##########
@@ -1078,12 +1081,16 @@ private void updateDisabledInstances(Collection<InstanceConfig> allInstanceConfi
// Move the calculating disabled instances to refresh
_disabledInstanceForPartitionMap.clear();
_disabledInstanceSet.clear();
+ _disabledInstancesForAllPartitionsSet.clear();
for (InstanceConfig config : allInstanceConfigs) {
Map<String, List<String>> disabledPartitionMap = config.getDisabledPartitionsMap();
- if (config.getInstanceOperation().getOperation()
- .equals(InstanceConstants.InstanceOperation.DISABLE)) {
+ // Treat instance as disabled if it has "DISABLE" operation or "ALL_RESOURCES" in the disabled partition map
+ if (config.getInstanceOperation().getOperation().equals(InstanceConstants.InstanceOperation.DISABLE)) {
_disabledInstanceSet.add(config.getInstanceName());
}
+ if (disabledPartitionMap.containsKey(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY)) {
+ _disabledInstancesForAllPartitionsSet.add(config.getInstanceName());
Review Comment:
@junkaixue moved this into the resource controller data provider.
the `disabledInstancesForAllPartitions` is only maintained int he resource controller provider
`getDisabledInstancesForPartition` is overriden in resource controller data , calls super method then applies the instances from the `disabledInstancesForAllPartitions`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add support for ALL_RESOURCES key to disabled partitions [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2848:
URL: https://github.com/apache/helix/pull/2848#discussion_r1690540858
##########
helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestInstanceOperation.java:
##########
@@ -1361,6 +1361,96 @@ public void testEvacuationWithOfflineInstancesInCluster() throws Exception {
dropTestDBs(ImmutableSet.of("TEST_DB3_DELAYED_CRUSHED", "TEST_DB4_DELAYED_WAGED"));
}
+ @Test(dependsOnMethods = "testEvacuationWithOfflineInstancesInCluster")
+ public void testEvacuateWithDisabledPartition() throws Exception {
+ System.out.println(
+ "START TestInstanceOperation.testEvacuateWithDisabledPartition() at " + new Date(
+ System.currentTimeMillis()));
+ StateTransitionCountStateModelFactory stateTransitionCountStateModelFactory = new StateTransitionCountStateModelFactory();
+ removeOfflineOrInactiveInstances();
+ String testCrushedDBName = "testEvacuateWithDisabledPartition_CRUSHED_DB0";
+ String testWagedDBName = "testEvacuateWithDisabledPartition_WAGED_DB1";
+ String toDisableThenEvacuateInstanceName = "disable_then_evacuate_host";
+ addParticipant(toDisableThenEvacuateInstanceName, stateTransitionCountStateModelFactory);
+ MockParticipantManager toDisableThenEvacuateParticipant = _participants.get(_participants.size() - 1);
+
+ List<String> testResources = Arrays.asList(testCrushedDBName, testWagedDBName);
+ createResourceWithDelayedRebalance(CLUSTER_NAME, testCrushedDBName, "MasterSlave",
+ PARTITIONS, REPLICA, REPLICA-1, 200000, CrushEdRebalanceStrategy.class.getName());
+ createResourceWithWagedRebalance(CLUSTER_NAME, testWagedDBName, "MasterSlave", PARTITIONS,
+ REPLICA, REPLICA-1);
+
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+ int upwardSTCountBeforeDisableThenEvacuate = stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter();
+ int downwardSTCountBeforeDisableThenEvacuate = stateTransitionCountStateModelFactory.getDownwardStateTransitionCounter();
+
+
+ InstanceConfig instanceConfig = _gSetupTool.getClusterManagementTool().getInstanceConfig(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName);
+ instanceConfig.setInstanceEnabledForPartition(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY, "", false);
+ _gSetupTool.getClusterManagementTool().setInstanceConfig(CLUSTER_NAME, toDisableThenEvacuateInstanceName, instanceConfig);
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+ // EV should not have disabled instance above the lowest state (OFFLINE)
+ verifier(() -> {
+ for (String resource : testResources) {
+ ExternalView ev = _gSetupTool.getClusterManagementTool().getResourceExternalView(CLUSTER_NAME, resource);
+ for (String partition : ev.getPartitionSet()) {
+ if (ev.getStateMap(partition).containsKey(toDisableThenEvacuateInstanceName) && !ev.getStateMap(partition).
+ get(toDisableThenEvacuateInstanceName).equals("OFFLINE")) {
+ return false;
+ }
+ }
+ }
+ return true;
+ }, 5000);
+
+ // Assert node received downward state transitions and no upward transitions
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
+ upwardSTCountBeforeDisableThenEvacuate, "Upward state transitions should not have been received");
+ Assert.assertTrue(stateTransitionCountStateModelFactory.getDownwardStateTransitionCounter() >
+ downwardSTCountBeforeDisableThenEvacuate, "Should have received downward state transitions");
+
+ _gSetupTool.getClusterManagementTool().setInstanceOperation(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName, InstanceConstants.InstanceOperation.EVACUATE);
+
+ verifier(() -> _admin.isEvacuateFinished(CLUSTER_NAME, toDisableThenEvacuateInstanceName), 30000);
+
+ // Assert node received no upward state transitions after evacuation was called on already disabled node
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
+ upwardSTCountBeforeDisableThenEvacuate, "Upward state transitions should not have been received");
+
+ // Re-enable all partitions for the instance
+ instanceConfig = _gSetupTool.getClusterManagementTool().getInstanceConfig(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName);
+ instanceConfig.setInstanceEnabledForPartition(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY, "", true);
+ _gSetupTool.getClusterManagementTool().setInstanceConfig(CLUSTER_NAME, toDisableThenEvacuateInstanceName, instanceConfig);
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+
+ // Assert node received no upward state transitions after re-enabled partitions
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
+ upwardSTCountBeforeDisableThenEvacuate, "Upward state transitions should not have been received");
+
+ // Disable all partitions for the instance again
+ instanceConfig = _gSetupTool.getClusterManagementTool().getInstanceConfig(CLUSTER_NAME,
+ toDisableThenEvacuateInstanceName);
+ instanceConfig.setInstanceEnabledForPartition(InstanceConstants.ALL_RESOURCES_DISABLED_PARTITION_KEY, "", true);
+ _gSetupTool.getClusterManagementTool().setInstanceConfig(CLUSTER_NAME, toDisableThenEvacuateInstanceName, instanceConfig);
+ Assert.assertTrue(_clusterVerifier.verifyByPolling());
+
+ // Assert node received no upward state transitions after disabling already evacuated node
+ Assert.assertEquals(stateTransitionCountStateModelFactory.getUpwardStateTransitionCounter(),
Review Comment:
Added assertion that no further downward state transits are experienced when you disable ALL_RESOURCES on an already evacuated node
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org