You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/05/26 03:45:37 UTC
[GitHub] [helix] xyuanlu opened a new pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
xyuanlu opened a new pull request #1756:
URL: https://github.com/apache/helix/pull/1756
### Issues
- [X] My PR addresses the following Helix issues and references them in the PR description:
#1755
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
ReadOnlyWagedRebalancer should not require a was Zk address in constructor.
In a multi ZK scenario, the address might be a mock address or a fake one. Passing a raw ZK address does not make sense.
### Tests
- [X] The following tests are written for this issue:
This API will be called by a REST API added in PR #1747. Unit test added in that PR will cover this change.
- The following is the result of the "mvn test" command on the appropriate module:
(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)
### Changes that Break Backward Compatibility (Optional)
- My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
(Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] junkaixue merged pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
junkaixue merged pull request #1756:
URL: https://github.com/apache/helix/pull/1756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640283326
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -295,7 +352,6 @@ public static String serializeByComma(List<String> objects) {
LOG.error("getIdealAssignmentForWagedFullAuto(): Failed to compute ResourceAssignments!", e);
} finally {
// Close all ZK connections
- baseDataAccessor.close();
readOnlyWagedRebalancer.close();
Review comment:
For helix-reset, the caller create the accessor and passed in, right? So it is not created in this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640857364
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -278,6 +282,17 @@ public HelixDataAccessor getDataAccessor(String clusterName) {
return _byteArrayZkBaseDataAccessor;
}
+ public ZkBucketDataAccessor getZkBucketDataAccessor() {
Review comment:
Please make a clean change, just don't touch ServerContext
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640287876
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -278,6 +282,17 @@ public HelixDataAccessor getDataAccessor(String clusterName) {
return _byteArrayZkBaseDataAccessor;
}
+ public ZkBucketDataAccessor getZkBucketDataAccessor() {
Review comment:
Since this is not used right now, shall we put it in the next PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640100876
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -210,19 +231,36 @@ public static String serializeByComma(List<String> objects) {
String metadataStoreAddress, ClusterConfig clusterConfig,
List<InstanceConfig> instanceConfigs, List<String> liveInstances,
List<IdealState> idealStates, List<ResourceConfig> resourceConfigs) {
- return getAssignmentForWagedFullAutoImpl(metadataStoreAddress, clusterConfig,
- instanceConfigs, liveInstances, idealStates, resourceConfigs, false);
+ return getAssignmentForWagedFullAutoHelper(metadataStoreAddress, clusterConfig, instanceConfigs,
+ liveInstances, idealStates, resourceConfigs, false);
}
- /*
- * If usePrefLists is set to true, the returned assignment is based on preference lists; if
- * false, the returned assignment is based on partition state mapping, which may differ from
- * preference lists.
- */
- private static Map<String, ResourceAssignment> getAssignmentForWagedFullAutoImpl(
+ private static Map<String, ResourceAssignment> getAssignmentForWagedFullAutoHelper(
+ ZkBucketDataAccessor zkBucketDataAccessor, HelixDataAccessor helixDataAccessor,
+ ClusterConfig clusterConfig, List<InstanceConfig> instanceConfigs, List<String> liveInstances,
+ List<IdealState> idealStates, List<ResourceConfig> resourceConfigs, boolean usePrefLists) {
+
+ // Copy the cluster config and make globalRebalance happen synchronously
+ // Otherwise, globalRebalance may not complete and this util might end up returning
+ // an empty assignment.
+ ClusterConfig globalSyncClusterConfig = new ClusterConfig(clusterConfig.getRecord());
+ globalSyncClusterConfig.setGlobalRebalanceAsyncMode(false);
+
+ // Create an instance of read-only WAGED rebalancer
+ ReadOnlyWagedRebalancer readOnlyWagedRebalancer =
+ new ReadOnlyWagedRebalancer(zkBucketDataAccessor, globalSyncClusterConfig.getClusterName(),
+ globalSyncClusterConfig.getGlobalRebalancePreference());
+
+ return getAssignmentForWagedFullAutoImpl(helixDataAccessor, readOnlyWagedRebalancer,
+ globalSyncClusterConfig, instanceConfigs, liveInstances, idealStates, resourceConfigs,
+ usePrefLists);
+ }
+
+ private static Map<String, ResourceAssignment> getAssignmentForWagedFullAutoHelper(
Review comment:
TFTR. If we make data accessor construction logic private, when adding a new REAT API, then the caller still need to pass a raw ZK address?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640045956
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/ReadOnlyWagedRebalancer.java
##########
@@ -69,6 +71,10 @@ public void updateChangeDetectorSnapshots(ResourceControllerDataProvider dataPro
super(new ZkBucketDataAccessor(metadataStoreAddress), clusterName);
}
+ ReadOnlyAssignmentMetadataStore(ZkBucketDataAccessor zkBucketDataAccessor, String clusterName) {
Review comment:
Optional, but if there is no other use cases, please just remove the older constructor. The newer one is more specific and avoid misusage, IMO.
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -295,7 +352,6 @@ public static String serializeByComma(List<String> objects) {
LOG.error("getIdealAssignmentForWagedFullAuto(): Failed to compute ResourceAssignments!", e);
} finally {
// Close all ZK connections
- baseDataAccessor.close();
readOnlyWagedRebalancer.close();
Review comment:
readOnlyWagedRebalancer is also passed in object, why closing it but removing baseDataAccessor close?
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -239,6 +277,25 @@ public static String serializeByComma(List<String> objects) {
new ReadOnlyWagedRebalancer(metadataStoreAddress, globalSyncClusterConfig.getClusterName(),
globalSyncClusterConfig.getGlobalRebalancePreference());
+ Map<String, ResourceAssignment> result =
+ getAssignmentForWagedFullAutoImpl(helixDataAccessor, readOnlyWagedRebalancer,
+ globalSyncClusterConfig, instanceConfigs, liveInstances, idealStates, resourceConfigs,
+ usePrefLists);
+ baseDataAccessor.close();
Review comment:
close readOnlyWagedRebalancer?
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -210,19 +231,36 @@ public static String serializeByComma(List<String> objects) {
String metadataStoreAddress, ClusterConfig clusterConfig,
List<InstanceConfig> instanceConfigs, List<String> liveInstances,
List<IdealState> idealStates, List<ResourceConfig> resourceConfigs) {
- return getAssignmentForWagedFullAutoImpl(metadataStoreAddress, clusterConfig,
- instanceConfigs, liveInstances, idealStates, resourceConfigs, false);
+ return getAssignmentForWagedFullAutoHelper(metadataStoreAddress, clusterConfig, instanceConfigs,
+ liveInstances, idealStates, resourceConfigs, false);
}
- /*
- * If usePrefLists is set to true, the returned assignment is based on preference lists; if
- * false, the returned assignment is based on partition state mapping, which may differ from
- * preference lists.
- */
- private static Map<String, ResourceAssignment> getAssignmentForWagedFullAutoImpl(
+ private static Map<String, ResourceAssignment> getAssignmentForWagedFullAutoHelper(
+ ZkBucketDataAccessor zkBucketDataAccessor, HelixDataAccessor helixDataAccessor,
+ ClusterConfig clusterConfig, List<InstanceConfig> instanceConfigs, List<String> liveInstances,
+ List<IdealState> idealStates, List<ResourceConfig> resourceConfigs, boolean usePrefLists) {
+
+ // Copy the cluster config and make globalRebalance happen synchronously
+ // Otherwise, globalRebalance may not complete and this util might end up returning
+ // an empty assignment.
+ ClusterConfig globalSyncClusterConfig = new ClusterConfig(clusterConfig.getRecord());
+ globalSyncClusterConfig.setGlobalRebalanceAsyncMode(false);
+
+ // Create an instance of read-only WAGED rebalancer
+ ReadOnlyWagedRebalancer readOnlyWagedRebalancer =
+ new ReadOnlyWagedRebalancer(zkBucketDataAccessor, globalSyncClusterConfig.getClusterName(),
+ globalSyncClusterConfig.getGlobalRebalancePreference());
+
+ return getAssignmentForWagedFullAutoImpl(helixDataAccessor, readOnlyWagedRebalancer,
+ globalSyncClusterConfig, instanceConfigs, liveInstances, idealStates, resourceConfigs,
+ usePrefLists);
+ }
+
+ private static Map<String, ResourceAssignment> getAssignmentForWagedFullAutoHelper(
Review comment:
This method is making the code here more complicated. I suggest not adding this one but making data accessor construction logic private methods and then call them repeatedly in the existing APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640223491
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -295,7 +352,6 @@ public static String serializeByComma(List<String> objects) {
LOG.error("getIdealAssignmentForWagedFullAuto(): Failed to compute ResourceAssignments!", e);
} finally {
// Close all ZK connections
- baseDataAccessor.close();
readOnlyWagedRebalancer.close();
Review comment:
If we pass a baseDataAccessor created by caller, then that accessor should not be closed here. readOnlyWagedRebalancer is created in this function so we close it in final block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640287390
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -295,7 +352,6 @@ public static String serializeByComma(List<String> objects) {
LOG.error("getIdealAssignmentForWagedFullAuto(): Failed to compute ResourceAssignments!", e);
} finally {
// Close all ZK connections
- baseDataAccessor.close();
readOnlyWagedRebalancer.close();
Review comment:
I see, I was thinking about the zkBucketDataAccessor. It is not closed when the rebalancer is closed.
You are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1756:
URL: https://github.com/apache/helix/pull/1756#issuecomment-849932748
This PR is ready to be merged. Approved by @jiajunwang.
commit message.
Add a Waged rebalancer util api that do not need raw zk address
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640290524
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -278,6 +282,17 @@ public HelixDataAccessor getDataAccessor(String clusterName) {
return _byteArrayZkBaseDataAccessor;
}
+ public ZkBucketDataAccessor getZkBucketDataAccessor() {
Review comment:
Yea make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1756: Add a Waged rebalancer util api that do not need raw zk address.
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1756:
URL: https://github.com/apache/helix/pull/1756#discussion_r640284883
##########
File path: helix-core/src/main/java/org/apache/helix/util/HelixUtil.java
##########
@@ -295,7 +352,6 @@ public static String serializeByComma(List<String> objects) {
LOG.error("getIdealAssignmentForWagedFullAuto(): Failed to compute ResourceAssignments!", e);
} finally {
// Close all ZK connections
- baseDataAccessor.close();
readOnlyWagedRebalancer.close();
Review comment:
Yes this is correct. So we should not close accessor here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org