You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "zpinto (via GitHub)" <gi...@apache.org> on 2025/04/02 21:19:36 UTC
[PR] Improve setInstanceOperation preformance [helix]
zpinto opened a new pull request, #3017:
URL: https://github.com/apache/helix/pull/3017
### Issues
- [x] Currently setInstanceOperation takes a long time because we are unecessarily finding instances with matching logicalIds as well as individually getting all instance configs in the cluster.
(#200 - Link your issue number here: You can write "Fixes #XXX". Please use the proper keyword so that the issue gets closed automatically. See https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue
Any of the following keywords can be used: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved)
### Description
This change will switch to parallel get on all instance configs and avoid calling findInstancesWithMatchingLogicalId for instance operation transitions where this check is not required.
### Tests
Current tests cover this change, as the result/response of these APIs will not change.
### Changes that Break Backward Compatibility (Optional)
- My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
(Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Improve setInstanceOperation performance [helix]
Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on PR #3017:
URL: https://github.com/apache/helix/pull/3017#issuecomment-2802925766
This PR is ready to be merged.
Commit Message:
```
This change will switch to parallel/async get on all instance configs, using HelixDataAccessor, and avoid calling findInstancesWithMatchingLogicalId for instance operation transitions where this check is not required.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Improve setInstanceOperation performance [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #3017:
URL: https://github.com/apache/helix/pull/3017#discussion_r2027724281
##########
helix-core/src/main/java/org/apache/helix/util/InstanceUtil.java:
##########
@@ -148,17 +201,57 @@ public static List<InstanceConfig> findInstancesWithMatchingLogicalId(
.collect(Collectors.toList());
}
+ /**
+ * Finds the instances that have a matching logical ID with the given instance.
+ *
+ * @param clusterName The cluster name
+ * @param instanceConfig The instance configuration to match
+ * @return A list of matching instances
+ */
+ public static List<InstanceConfig> findInstancesWithMatchingLogicalId(
+ BaseDataAccessor<ZNRecord> baseDataAccessor, String clusterName,
+ InstanceConfig instanceConfig) {
+ HelixDataAccessor helixDataAccessor = new ZKHelixDataAccessor(clusterName, baseDataAccessor);
+
+ ClusterConfig clusterConfig =
+ helixDataAccessor.getProperty(helixDataAccessor.keyBuilder().clusterConfig());
+ String logicalIdKey =
+ ClusterTopologyConfig.createFromClusterConfig(clusterConfig).getEndNodeType();
+
+ List<InstanceConfig> instanceConfigs =
+ helixDataAccessor.getChildValues(helixDataAccessor.keyBuilder().instanceConfigs(), true);
+
+ // Retrieve and filter instances with matching logical ID
+ return instanceConfigs.stream().filter(potentialInstanceConfig ->
+ !potentialInstanceConfig.getInstanceName().equals(instanceConfig.getInstanceName())
+ && potentialInstanceConfig.getLogicalId(logicalIdKey)
+ .equals(instanceConfig.getLogicalId(logicalIdKey))).collect(Collectors.toList());
+ }
+
+ private static List<InstanceConfig> findInstancesWithMatchingLogicalId(
+ @Nullable BaseDataAccessor<ZNRecord> baseDataAccessor,
+ @Nullable ConfigAccessor configAccessor, String clusterName, InstanceConfig instanceConfig) {
+ if (baseDataAccessor != null) {
+ return findInstancesWithMatchingLogicalId(baseDataAccessor, clusterName, instanceConfig);
+ } else if (configAccessor != null) {
+ return findInstancesWithMatchingLogicalId(configAccessor, clusterName, instanceConfig);
+ } else {
+ throw new HelixException(
+ "Both BaseDataAccessor and ConfigAccessor cannot be null at the same time");
+ }
Review Comment:
A better way to make the code more clear is:
if (baseDataAccessor == null && configAccessor == null) {
throw new HelixException(
"Both BaseDataAccessor and ConfigAccessor cannot be null at the same time");
}
return findInstancesWithMatchingLogicalId(baseDataAccessor != null ? baseDataAccessor : configAccessor, clusterName, instanceConfig);
This simplify the branch of coding and also return the fail over logic at first beginning with out further checks. Especially when you have function with a lot of arguments repeatedly shows up, readability can be reduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Improve setInstanceOperation performance [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #3017:
URL: https://github.com/apache/helix/pull/3017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Improve setInstanceOperation performance [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on PR #3017:
URL: https://github.com/apache/helix/pull/3017#issuecomment-2774201205
This will close: #2928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Improve setInstanceOperation performance [helix]
Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on code in PR #3017:
URL: https://github.com/apache/helix/pull/3017#discussion_r2032194068
##########
helix-core/src/main/java/org/apache/helix/util/InstanceUtil.java:
##########
@@ -148,17 +201,57 @@ public static List<InstanceConfig> findInstancesWithMatchingLogicalId(
.collect(Collectors.toList());
}
+ /**
+ * Finds the instances that have a matching logical ID with the given instance.
+ *
+ * @param clusterName The cluster name
+ * @param instanceConfig The instance configuration to match
+ * @return A list of matching instances
+ */
+ public static List<InstanceConfig> findInstancesWithMatchingLogicalId(
+ BaseDataAccessor<ZNRecord> baseDataAccessor, String clusterName,
+ InstanceConfig instanceConfig) {
+ HelixDataAccessor helixDataAccessor = new ZKHelixDataAccessor(clusterName, baseDataAccessor);
+
+ ClusterConfig clusterConfig =
+ helixDataAccessor.getProperty(helixDataAccessor.keyBuilder().clusterConfig());
+ String logicalIdKey =
+ ClusterTopologyConfig.createFromClusterConfig(clusterConfig).getEndNodeType();
+
+ List<InstanceConfig> instanceConfigs =
+ helixDataAccessor.getChildValues(helixDataAccessor.keyBuilder().instanceConfigs(), true);
+
+ // Retrieve and filter instances with matching logical ID
+ return instanceConfigs.stream().filter(potentialInstanceConfig ->
+ !potentialInstanceConfig.getInstanceName().equals(instanceConfig.getInstanceName())
+ && potentialInstanceConfig.getLogicalId(logicalIdKey)
+ .equals(instanceConfig.getLogicalId(logicalIdKey))).collect(Collectors.toList());
+ }
+
+ private static List<InstanceConfig> findInstancesWithMatchingLogicalId(
+ @Nullable BaseDataAccessor<ZNRecord> baseDataAccessor,
+ @Nullable ConfigAccessor configAccessor, String clusterName, InstanceConfig instanceConfig) {
+ if (baseDataAccessor != null) {
+ return findInstancesWithMatchingLogicalId(baseDataAccessor, clusterName, instanceConfig);
+ } else if (configAccessor != null) {
+ return findInstancesWithMatchingLogicalId(configAccessor, clusterName, instanceConfig);
+ } else {
+ throw new HelixException(
+ "Both BaseDataAccessor and ConfigAccessor cannot be null at the same time");
+ }
Review Comment:
Thanks for the review, I have incorporated the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Improve setInstanceOperation performance [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on PR #3017:
URL: https://github.com/apache/helix/pull/3017#issuecomment-2797805900
Had an offline discussion about wether we want to use multistream with configAccessor or using dataAccessor.
if we use multistream with configAccessor, we need to follow the pattern for all current usage of configAccessor - call `ZKUtil.isInstanceSetup` check before calling configAccessor. It is not efficient as inside configAccessor, it is `ZKUtil.isInstanceSetup` called again, resulting in many duplicate ZK read.
If we change to using dataAccessor, we need to deprecate the prev API and create a new one.
Considering we need to get all instances config in the cluster, I think we could favor performance to code style. And also do code clean in follow up changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org