You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/05/30 00:37:07 UTC
[GitHub] [helix] xyuanlu opened a new pull request #1043: Code clean up Topology.java
xyuanlu opened a new pull request #1043:
URL: https://github.com/apache/helix/pull/1043
### Issues
- [X] My PR addresses the following Helix issues and references them in the PR description:
#1042 Validate instance topology configuration before let it comes online
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
We need sanity check for topology definition in InstanceConfig in multiple places. Since we already have sanity check logic in controller/rebalancer/topology/Topology.java and controller/rebalancer/waged/model/AssignableNode.java, it's better to extract the logic into an single util function first and use the same logic to do sanity check in multiple places.
This PR is the first PR for issue #1042. It extracts the sanity check in multiple functions into one single function in Topology.java to align with the topology verification logic in AssignableNode.java.
There is one minor behavior change compared to the old version: If there is a key in InstanceConfig.DOMAIN not defined in clusterConfig.topology, we skip the key silently same as how its done in AssignableNode.java. In the old code, we used to add a
warning in log.
### Tests
- [X] The following tests are written for this issue:
No new test is added since this PR is code clean up.
- [X] The following is the result of the "mvn test" command on the appropriate module:
`[ERROR] Failures:
[ERROR] org.apache.helix.integration.TestDisablePartition.testDisableFullAuto(org.apache.helix.integration.TestDisablePartition)
[ERROR] Run 1: TestDisablePartition.testDisableFullAuto:202 expected:<OFFLINE> but was:<STANDBY>
[INFO] Run 2: PASS
[INFO] [ERROR] TestControllerLeadershipChange.testMissingTopStateDurationMonitoring:262 expected:<true> but was:<false>
[ERROR] TestStateTransitionTimeout.testStateTransitionTimeOut:166 expected:<true> but was:<false>
[ERROR] TestStateTransitionTimeoutWithResource.testStateTransitionTimeOut:174 expected:<true> but was:<false>
[INFO]
[ERROR] Tests run: 1147, Failures: 4, Errors: 0, Skipped: 1
[INFO]
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.198 s - in org.apache.helix.integration.controller.TestControllerLeadershipChange
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.333 s - in org.apache.helix.integration.paticipant.TestStateTransitionTimeout
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.97 s - in org.apache.helix.integration.paticipant.TestStateTransitionTimeoutWithResource
[INFO]
`
### Commits
- [X] My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation (Optional)
- [X] In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Code Quality
- [X] My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438990633
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
Review comment:
update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r442474680
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +191,122 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ try {
+ LinkedHashMap<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys)
+ throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap = new HashMap<>();
Review comment:
You can check once you get the map in the same way. Please try to avoid duplicate logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434989030
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -55,16 +55,10 @@
private final List<String> _liveInstances;
private final Map<String, InstanceConfig> _instanceConfigMap;
private final ClusterConfig _clusterConfig;
- private final boolean _topologyAwareEnabled;
+ private static final String DEFAULT_PATH_PREFIX = "Helix_default_";
Review comment:
TFTR.
I kept the same way as it is. I think we probably should not change the default behavior when doing code clean up.
In old code line 105:
104 for (String type : _types) {
105 _defaultDomainPathValues.put(type, "Helix_default_" + type);
106 lastType = type;
107 }
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-646917678
Just noticed one more issue. Please check.
Moreover, please specify your final commit description. We will squash and merge all commits into one and use that description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443092207
##########
File path: helix-core/src/main/java/org/apache/helix/model/InstanceConfig.java
##########
@@ -150,12 +150,22 @@ public String getDomainAsString() {
*/
public Map<String, String> getDomainAsMap() {
String domain = getDomainAsString();
+ Map<String, String> domainAsMap = new HashMap<>();
if (domain == null || domain.isEmpty()) {
- return Collections.emptyMap();
+ return domainAsMap;
+ }
+
+ String[] pathPairs = domain.trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Domain-Value pair %s is not valid.", pair));
+ }
+ domainAsMap.put(values[0], values[1]);
Review comment:
Oh I got what you are saying. Yes thats correct. Will update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443091509
##########
File path: helix-core/src/main/java/org/apache/helix/model/InstanceConfig.java
##########
@@ -150,12 +150,22 @@ public String getDomainAsString() {
*/
public Map<String, String> getDomainAsMap() {
String domain = getDomainAsString();
+ Map<String, String> domainAsMap = new HashMap<>();
if (domain == null || domain.isEmpty()) {
- return Collections.emptyMap();
+ return domainAsMap;
+ }
+
+ String[] pathPairs = domain.trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Domain-Value pair %s is not valid.", pair));
+ }
+ domainAsMap.put(values[0], values[1]);
Review comment:
I think if the value or key is null, then this pair shouldn't be count as a valid input.
Since currently the callers of this function only checks if key exists in the returned map and uses the corresponding value.
`domainAsMap.getOrDefault(key, "Default_" + key`
So I think the user all assume that all k-v pairs in returned map are valid (not null).
Thus, I think we should throw exception for empty value. (Original version already handles no splitter and null key errors).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439067010
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
Review comment:
If your later usage assumes anything about the iteration order, then we should make this Map explicitly an ordered one.
And addEndNode() will require an ordered map. I suggest you change the addEndNode() signature so others won't be able to pass any non-ordered map. With that change, you have to change this Map to LinkedHashMap here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-646880040
This PR is ready to be merged, approved by @jiajunwang
Code clean up -- Topology.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434989142
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,42 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ for (; lastValidTypeIdx >= 0; --lastValidTypeIdx) {
Review comment:
Will update.
THX
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433520034
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
Review comment:
Yes, what you said about using the interface is on point.
I understand where you are coming from - you want to make it explicit that the parameter that's being passed in is ordered. But let's revisit what I said above - making the declaration LinkedHashMap does not help you in that case.
LinkedHashMap's ordering is not deterministic. It does not enforce the right ordering you need in the map. So you aren't really enforcing anything there other than the map will have some order.
If you want to enforce some sort of ordering of elements, you should sort internally. Say you need it sorted by alphabetical order, then you should convert this map into a TreeMap.
Alternatively, stating that the topology map must be ordered in some way is also acceptable in JavaDoc, but in that case, the right thing to do is to be able to detect if the ordering has a problem, be able to throw an exception of some sort, alerting the user.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439688790
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +78,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
+ _clusterTopologyKeys = new LinkedHashSet<>();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int lastValidTypeIdx = 0;
+ for (int i = 0; i < topologyKeys.length; i++) {
+ if (topologyKeys[i].length() != 0) {
+ _clusterTopologyKeys.add(topologyKeys[i]);
+ lastValidTypeIdx = i;
}
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (_clusterTopologyKeys.size() == 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyKeys[lastValidTypeIdx];
Review comment:
I am not sure if there is a get() function for LinkedHashSet.
https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html#method.summary
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu edited a comment on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu edited a comment on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-643568995
I ran a CPU profiler for my previous version (34bf638). It shows the lamda function
`defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key)); `
and `InstanceConfig.getDomaiAsMap` are two hotspots for CPU consumption. I changed them back to the original implementation.
After these changes, the new CPU profiler result of running testCreateClusterTopology is around 2%, comparing to
2 - 3% for the original code and 6% with previous version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu edited a comment on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu edited a comment on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-643568995
I ran a CPU profiler for my previous version (34bf638). It shows the lamda function
`defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key)); `
and InstanceConfig.getDomaiAsMap are two hotspots for CPU consumption. I changed them back to the original implementation.
After these changes, the new result of running testCreateClusterTopology is ~2%, comparing to
2~3% for the original code and 6% with previous version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439647009
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
+ if (topologyStr[lastValidTypeIdx].length() != 0) {
+ break;
}
+ --lastValidTypeIdx;
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (lastValidTypeIdx < 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyStr[lastValidTypeIdx];
+ _faultZoneType = clusterConfig.getFaultZoneType();
+ if (_faultZoneType == null) {
+ _faultZoneType = _endNodeType;
+ }
+ if (Arrays.stream(topologyStr).noneMatch(type -> type.equals(_faultZoneType))) {
Review comment:
Had an offline discussion. Although the time consumed here in the constructor is the same, it is better to have a smaller loop with no empty string when creating the domain k-v map for all instance. Having extra empty string in the for loop may introduce performance issue considering the number of instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432891174
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
Please correct me if I am wrong. I think Map does not guarantee the order in which pairs were inserted into the map. When constructing the topology tree, it is important to maintain the original order for the path.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438481995
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
+ if (topologyStr[lastValidTypeIdx].length() != 0) {
+ break;
}
+ --lastValidTypeIdx;
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (lastValidTypeIdx < 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyStr[lastValidTypeIdx];
+ _faultZoneType = clusterConfig.getFaultZoneType();
+ if (_faultZoneType == null) {
+ _faultZoneType = _endNodeType;
+ }
+ if (Arrays.stream(topologyStr).noneMatch(type -> type.equals(_faultZoneType))) {
Review comment:
This operation is more expansive than the original code. What's the purpose of modifying this part? It does not seem to be simpler.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -55,16 +55,10 @@
private final List<String> _liveInstances;
private final Map<String, InstanceConfig> _instanceConfigMap;
private final ClusterConfig _clusterConfig;
- private final boolean _topologyAwareEnabled;
+ private static final String DEFAULT_PATH_PREFIX = "Helix_default_";
Review comment:
Why it is PATH_PREFIX, I think it should be the VALUE_PREFIX. Of course, the value prefix is not a good name either. Let's call it DEFAULT_DOMAIN_PREFIX. And add some comments to indicate what's the usage.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
+ if (topologyStr[lastValidTypeIdx].length() != 0) {
+ break;
}
+ --lastValidTypeIdx;
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (lastValidTypeIdx < 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyStr[lastValidTypeIdx];
+ _faultZoneType = clusterConfig.getFaultZoneType();
+ if (_faultZoneType == null) {
+ _faultZoneType = _endNodeType;
+ }
+ if (Arrays.stream(topologyStr).noneMatch(type -> type.equals(_faultZoneType))) {
Review comment:
Note this operation is called many times during the rebalance, so any incremental delay is impactful. Must be very careful, or the rebalance latency will be increased.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fail the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
+ }
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int numOfmatchedKeys = 0;
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value = DEFAULT_PATH_PREFIX + key;
+ } else {
+ numOfmatchedKeys++;
+ }
+ instanceTopologyMap.put(key, value);
+ }
+ }
+ if (numOfmatchedKeys != domainAsMap.size()) {
+ logger.warn(
+ "Key-value pairs in InstanceConfig.Domain {} do not align with keys in ClusterConfig.Topology {}",
+ domain, topologyKeys.toString());
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return instanceTopologyMap;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
+ * Input param 'pathNameMap' must have a certain order where the order of the keys should be
Review comment:
In this case, please convert the pathNameMap to be a LinkedHashMap input.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
Review comment:
This logic cannot handle the following case correctly,
"/domain/rack///node/"
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
Review comment:
Why not directly call domainAsMap = instanceConfig.getDomainAsMap();?
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
Review comment:
This comment is not right.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
Review comment:
This should be LinkedHashMap
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
Review comment:
instance -> instanceName
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fail the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
+ }
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
Review comment:
Could you just input the String[] topologyStr (or a LinkedHashSet which I would recommend) to this method so you don't do the calculation again?
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fail the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
+ }
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int numOfmatchedKeys = 0;
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value = DEFAULT_PATH_PREFIX + key;
+ } else {
+ numOfmatchedKeys++;
+ }
+ instanceTopologyMap.put(key, value);
+ }
+ }
+ if (numOfmatchedKeys != domainAsMap.size()) {
+ logger.warn(
+ "Key-value pairs in InstanceConfig.Domain {} do not align with keys in ClusterConfig.Topology {}",
Review comment:
Mention here that the default domain value has been used. Otherwise, this warning looks like a major problem.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
Review comment:
I would prefer to keep it. A LinkedHashSet or ArrayList would be much easier to handle compared with array.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
Review comment:
This is duplicate code, let's have a private method to check if the node is enabled or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432891188
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
Review comment:
Same as above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443091790
##########
File path: helix-core/src/main/java/org/apache/helix/model/InstanceConfig.java
##########
@@ -150,12 +150,22 @@ public String getDomainAsString() {
*/
public Map<String, String> getDomainAsMap() {
String domain = getDomainAsString();
+ Map<String, String> domainAsMap = new HashMap<>();
if (domain == null || domain.isEmpty()) {
- return Collections.emptyMap();
+ return domainAsMap;
+ }
+
+ String[] pathPairs = domain.trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Domain-Value pair %s is not valid.", pair));
+ }
+ domainAsMap.put(values[0], values[1]);
Review comment:
Consider this case:
"a = b, c = d", the current logic will have a map of {"a ": " b", "c " : " d"} (notice the space), and what we expect is {"a": "b", "c" : "d"}.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433522629
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
Review comment:
This is an out parameter. So the order will be kept when k-v pairs are added to the LinkedHashMap in this function. Forcing the caller to pass in a LinkedHashMap will guarantee the order in which pairs were inserted into the map when user uses the map later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] dasahcc commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434979363
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -55,16 +55,10 @@
private final List<String> _liveInstances;
private final Map<String, InstanceConfig> _instanceConfigMap;
private final ClusterConfig _clusterConfig;
- private final boolean _topologyAwareEnabled;
+ private static final String DEFAULT_PATH_PREFIX = "Helix_default_";
Review comment:
Helix usually have all capitalized strings as our naming convention. Is this newly introduced or exists before?
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,42 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ for (; lastValidTypeIdx >= 0; --lastValidTypeIdx) {
Review comment:
Why not change this to be while clause since this already lost advantage of for loop for instantiation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439647050
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fail the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
+ }
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int numOfmatchedKeys = 0;
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value = DEFAULT_PATH_PREFIX + key;
+ } else {
+ numOfmatchedKeys++;
+ }
+ instanceTopologyMap.put(key, value);
+ }
+ }
+ if (numOfmatchedKeys != domainAsMap.size()) {
+ logger.warn(
+ "Key-value pairs in InstanceConfig.Domain {} do not align with keys in ClusterConfig.Topology {}",
+ domain, topologyKeys.toString());
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return instanceTopologyMap;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
+ * Input param 'pathNameMap' must have a certain order where the order of the keys should be
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433517300
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
Review comment:
I am planning to extract this function into a public util function. It's better to explicitly state that the order is important or maybe still use the general map interface with comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433522629
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
Review comment:
This is an out parameter. So the order will be kept when k-v pairs are added to the LinkedHashMap in this function. Forcing the caller to pass in a LinkedHashMap will guarantee the order in which pairs were inserted into the map when user use the map later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439650207
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
Review comment:
Updated.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
Review comment:
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu edited a comment on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu edited a comment on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-646880040
This PR is ready to be merged, approved by @jiajunwang
Code clean up -- Topology.java
This change did come code refactor work for Topology.java. It extracts the sanity check logic in multiple functions into one single function so it can be used else where as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu edited a comment on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu edited a comment on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-643568995
I ran a CPU profiler for my previous version (34bf638). It shows the lamda function
`defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key)); `
and `InstanceConfig.getDomaiAsMap` are two hotspots for CPU consumption. I changed them back to the original implementation.
After these changes, the new result of running testCreateClusterTopology is ~2%, comparing to
2~3% for the original code and 6% with previous version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438506585
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
+ if (topologyStr[lastValidTypeIdx].length() != 0) {
+ break;
}
+ --lastValidTypeIdx;
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (lastValidTypeIdx < 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyStr[lastValidTypeIdx];
+ _faultZoneType = clusterConfig.getFaultZoneType();
+ if (_faultZoneType == null) {
+ _faultZoneType = _endNodeType;
+ }
+ if (Arrays.stream(topologyStr).noneMatch(type -> type.equals(_faultZoneType))) {
Review comment:
One more optimization. We could put this check in the else branch for line 97. If we already set the _faultZoneType to be the same as _endNodeType, then _faultZoneType must belongs to topologyStr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r435629346
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
Review comment:
I changed the out param to a return value. It would be more clear and the definition in the caller can be changed to Map interface.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434215820
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -55,16 +55,10 @@
private final List<String> _liveInstances;
private final Map<String, InstanceConfig> _instanceConfigMap;
private final ClusterConfig _clusterConfig;
- private final boolean _topologyAwareEnabled;
+ private final String _defaultPathPrefix = "Helix_default_";
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-643568995
I ran a CPU profiler for my previous version (34bf638). It shows the lamda function and InstanceConfig.getDomaiAsMap are two hotspots for CPU consumption. I changed them back to the original implementation.
After these changes, the new result of running testCreateClusterTopology is ~2%, comparing to
2~3% for the original code and 6% with previous version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434215435
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439065862
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
+ if (topologyStr[lastValidTypeIdx].length() != 0) {
+ break;
}
+ --lastValidTypeIdx;
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (lastValidTypeIdx < 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyStr[lastValidTypeIdx];
+ _faultZoneType = clusterConfig.getFaultZoneType();
+ if (_faultZoneType == null) {
+ _faultZoneType = _endNodeType;
+ }
+ if (Arrays.stream(topologyStr).noneMatch(type -> type.equals(_faultZoneType))) {
Review comment:
Considering the case "/domain/rack///node/", you will need to process the array. So either you have a new array or a LinkedHashSet. The latter can save us some time here.
I was thinking about these 2 points together. And my conclusion is that keep the original logic is better : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433520992
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
If it's a private method, then you can explicitly state that in the JavaDoc and use Map to declare. We want to avoid unnecessary sorting/computation if we can be sure that the parameters are well-formed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432818347
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
Review comment:
Instead of `String.format`, I suggest parameterized logging: `logger.warn("Domain for instance {} is not set, ignore the instance!", instance);`
`String.format` is slower. And parameterized logging eliminates string concatenation if WARN level logging is not enabled.
Same for the other logging you changed in this PR.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
Review comment:
In what case the `IllegalArgumentException` is thrown from `getDomainAsMap`? Is this same behavior with the old condition check?
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
`LinkedHashMap` implements `Map` interface. I suggest keeping `Map`.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
Review comment:
I suggest making "Helix_default_" a constant because the constant eliminates redundant string objects: constant is only one string object but this would create one string each time.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
Review comment:
I suggest using `Map` interface as type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433002750
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
Review comment:
Use Map instead...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433541650
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
Review comment:
For this in class variable defined here in a private function for class constructor, i guess it is safe to assume the order will be maintained as expected since this private function has the full control of this object. Of course, more detailed comment/doc should be added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433527436
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
That make sense. Will update the pr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434989030
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -55,16 +55,10 @@
private final List<String> _liveInstances;
private final Map<String, InstanceConfig> _instanceConfigMap;
private final ClusterConfig _clusterConfig;
- private final boolean _topologyAwareEnabled;
+ private static final String DEFAULT_PATH_PREFIX = "Helix_default_";
Review comment:
TFTR.
I did not change the prefix. Please refer to line 105 in the old code.
104 for (String type : _types) {
105 _defaultDomainPathValues.put(type, "Helix_default_" + type);
106 lastType = type;
107 }
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439650267
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434215930
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433470812
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length()==0) {
+ value = _defaultPathPrefix + key;
+ }
+ instanceTopologyMap.put(key, value);
Review comment:
TFTR.
That's a good point. We probably should revisite the old sanity check logic and modify if needed.
One minor correcting. If the "domain" is set with a random value with no k-v split identifier found ('=' in this case), we will throw IllegalArgumentException.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
For private function like this, can we assume caller will strictly follow the instruction (of course with more detailed comment added) and guarantee the map has the right order? Or we could do one of the following to guarantee order:
1. pass in a separate array (or LinkedHashSet as line 63 in the old code) to keep the order of the key
2. add a lamda function to sorting internally.
THX
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
Review comment:
TFTR.
Please correct me if I am wrong. As a general rule, we're supposed use its interface in declaration rather than the actual implementation like the following.
`Map<String, String> instanceTopologyMap = new LinkedHashMap<>();`
The reason behind this is that the implementation of these interfaces is usually not relevant when handling them in the following logic. However, in my understanding, in some special cases like this, if we need a map when order is important, it is better to use a more specific interface or an implementation to explicitly state that the order is important. Or maybe still use the general interface with comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r434989030
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -55,16 +55,10 @@
private final List<String> _liveInstances;
private final Map<String, InstanceConfig> _instanceConfigMap;
private final ClusterConfig _clusterConfig;
- private final boolean _topologyAwareEnabled;
+ private static final String DEFAULT_PATH_PREFIX = "Helix_default_";
Review comment:
TFTR.
I kept the same way as it is. I think we probably should not change the default behavior when ding code clean up.
In old code line 105:
104 for (String type : _types) {
105 _defaultDomainPathValues.put(type, "Helix_default_" + type);
106 lastType = type;
107 }
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432893951
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
Review comment:
GetDomainAsMap will throw IllegalArgumentException if
1. The Splitter.on() param matches the empty string.
2. Has duplicate Key.
For example the controller does not know how to categorize the instance if user specified two different rack.
There is one behavior change (witch I should have documented in the pr description) is that, GetDomainAsMap won't throw exception if either key or value is an empty string. I think the new behavior is reasonable since
1. We ignore keys in DOMAIN that are not defined in ClusterConfig. When we have empty key, they should be treated as non-defined key. Instead of throw exception, it is ok to just ignore them. They are no harm to the topology tree anyway.
2. We will use default values for keys that are defined in ClusterConfig but missing InstanceConfig.DOMAIN. If there is an empty value defined, we should use the default value as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433003395
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
Let's use `Map`? Making the function signature LinkedHashMap doesn't necessarily guarantee the map has the right order.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443090524
##########
File path: helix-core/src/main/java/org/apache/helix/model/InstanceConfig.java
##########
@@ -150,12 +150,22 @@ public String getDomainAsString() {
*/
public Map<String, String> getDomainAsMap() {
String domain = getDomainAsString();
+ Map<String, String> domainAsMap = new HashMap<>();
if (domain == null || domain.isEmpty()) {
- return Collections.emptyMap();
+ return domainAsMap;
+ }
+
+ String[] pathPairs = domain.trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Domain-Value pair %s is not valid.", pair));
+ }
+ domainAsMap.put(values[0], values[1]);
Review comment:
We shall trim at the final values level but not the above layers. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r435634553
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length()==0) {
+ value = _defaultPathPrefix + key;
+ }
+ instanceTopologyMap.put(key, value);
Review comment:
There will be a following PR for this issue. May I suggest using this PR solely for code clean up? We probably should schedule a discussion to revisit the old sanity check logic and modify if needed.
THX
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length()==0) {
+ value = _defaultPathPrefix + key;
+ }
+ instanceTopologyMap.put(key, value);
Review comment:
TFTR.
That's a good point. We probably should revisit the old sanity check logic and modify if needed.
One minor correcting. If the "domain" is set with a random value with no k-v split identifier found ('=' in this case), we will throw IllegalArgumentException.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r442400834
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +191,122 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ try {
+ LinkedHashMap<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys)
+ throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap = new HashMap<>();
+ String[] pathPairs = instanceConfig.getDomainAsString().trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(String.format(
+ "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
+ pair, instanceName));
+ }
+ domainAsMap.put(values[0], values[1]);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ int numOfMatchedKeys = 0;
+ for (String key : clusterTopologyKeys) {
+ // if a key does not exist in the instance domain config, using the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value = _defaultDomainPathValues.get(key);
+ //defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key));
Review comment:
Forgot to remove the comment. Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439650077
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432891174
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
TFTR.
Please correct me if I am wrong. I think Map does not guarantee the order in which pairs were inserted into the map. When constructing the topology tree, it is important to maintain the original order for the path.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439709103
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String
+ .format("Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ instanceConfig.getDomainAsString(), instanceName), e);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ if (domainAsMap == null || domainAsMap.isEmpty()) {
+ throw new IllegalArgumentException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
+ }
+ int numOfMatchedKeys = 0;
+ for (String key : clusterTopologyKeys) {
+ // if a key does not exist in the instance domain config, using the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value =
+ defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key));
Review comment:
I changed it back to initialing the map in the constrictor. CPU profiler shows the overhead of having a lamda function here is too heavy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438990633
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
Review comment:
update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439650362
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fail the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
+ }
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
Review comment:
Updated.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ // we have the hierarchy style of domain id for instance.
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing ZONE_ID information, fail the rebalance.
+ throw new HelixException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing ZONE setting, ignore it should be fine.
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fail the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return null;
+ }
+ }
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int numOfmatchedKeys = 0;
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value = DEFAULT_PATH_PREFIX + key;
+ } else {
+ numOfmatchedKeys++;
+ }
+ instanceTopologyMap.put(key, value);
+ }
+ }
+ if (numOfmatchedKeys != domainAsMap.size()) {
+ logger.warn(
+ "Key-value pairs in InstanceConfig.Domain {} do not align with keys in ClusterConfig.Topology {}",
Review comment:
Updated.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig, instance, insConfig);
+ if (instanceTopologyMap != null) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
}
- pathValueMap.put(Types.ZONE.name(), zone);
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
}
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @param clusterConfig clusterConfig of the cluster.
+ * @param instance Name of the given instance.
+ * @param instanceConfig instanceConfig of the instance.
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(ClusterConfig clusterConfig,
+ String instance, InstanceConfig instanceConfig) {
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu edited a comment on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu edited a comment on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-643568995
I ran a CPU profiler for my previous version (34bf638). It shows the lamda function
`defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key)); `
and `InstanceConfig.getDomaiAsMap` are two hotspots for CPU consumption. I changed them back to the original implementation.
After these changes, the new result of running testCreateClusterTopology is around 2%, comparing to
2 - 3% for the original code and 6% with previous version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443058192
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +191,122 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ try {
+ LinkedHashMap<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys)
+ throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap = new HashMap<>();
Review comment:
As our offline discussion. I updated the InstanceConfig.getDomainAsMap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443091509
##########
File path: helix-core/src/main/java/org/apache/helix/model/InstanceConfig.java
##########
@@ -150,12 +150,22 @@ public String getDomainAsString() {
*/
public Map<String, String> getDomainAsMap() {
String domain = getDomainAsString();
+ Map<String, String> domainAsMap = new HashMap<>();
if (domain == null || domain.isEmpty()) {
- return Collections.emptyMap();
+ return domainAsMap;
+ }
+
+ String[] pathPairs = domain.trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Domain-Value pair %s is not valid.", pair));
+ }
+ domainAsMap.put(values[0], values[1]);
Review comment:
I think if the value or key is null, then this pair shouldn't be count as a valid input.
Since currently the callers of this function only checks if key exists in the returned map and uses the corresponding value. So i think the user all assume all k-v pairs in returned map are valid.
Thus, I think we should throw exception for empty value. (Original version already handles no splitter and null key errors).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433001675
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -55,16 +55,10 @@
private final List<String> _liveInstances;
private final Map<String, InstanceConfig> _instanceConfigMap;
private final ClusterConfig _clusterConfig;
- private final boolean _topologyAwareEnabled;
+ private final String _defaultPathPrefix = "Helix_default_";
Review comment:
Please review the convention for defining constant Strings in Java.
https://stackoverflow.com/questions/9639007/defining-constant-string-in-java
With that said, is this prefix necessary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432895092
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
Review comment:
TFTR. Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438510939
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
Review comment:
Shouldn't we use interface for declaration (as previous comments suggested)?
Or in this specific case it's better to explicitly state the ordering is important?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-646919874
TFTR. Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r441911201
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +191,122 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ try {
+ LinkedHashMap<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys)
+ throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap = new HashMap<>();
+ String[] pathPairs = instanceConfig.getDomainAsString().trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(String.format(
+ "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
+ pair, instanceName));
+ }
+ domainAsMap.put(values[0], values[1]);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ int numOfMatchedKeys = 0;
+ for (String key : clusterTopologyKeys) {
+ // if a key does not exist in the instance domain config, using the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value = _defaultDomainPathValues.get(key);
+ //defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key));
Review comment:
Remove
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +191,122 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ try {
+ LinkedHashMap<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys)
+ throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap = new HashMap<>();
Review comment:
Isn't this block the same logic as getDomainAsMap()?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438505561
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
+ if (topologyStr[lastValidTypeIdx].length() != 0) {
+ break;
}
+ --lastValidTypeIdx;
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (lastValidTypeIdx < 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyStr[lastValidTypeIdx];
+ _faultZoneType = clusterConfig.getFaultZoneType();
+ if (_faultZoneType == null) {
+ _faultZoneType = _endNodeType;
+ }
+ if (Arrays.stream(topologyStr).noneMatch(type -> type.equals(_faultZoneType))) {
Review comment:
Please correct me if I am wrong.
Comparing to previous approach (set. contains()), the stream.noneMatch() does introduced a o(n)worst case complexity.
However, since there is no new LinkedHashSet created, it saved a O(n) time for constructing the LinkedHashSet and a O(n) space complexity for the object itself.
It is better to pass the already parsed array to computeInstanceTopologyMap so we do not compute again in line 274. However, I think we could just use the original array topologyStr, no need to introduce another LinkedHashSet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433002690
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
Review comment:
@xyuanlu
Let's use a Map to declare?
And what you said about LinkedHashMap is true, but Map is a higher-level construct (interface) and LinkedHashMap is an implementation. Even if you wanted a LinkedHashMap instance, how do you guarantee that the map has the right order? If you want to be 100% sure about the order, you have two options:
1. Make it crystal clear that the user must pass in ordered map (TreeMap or LinkedHashMap)
2. do your own sorting internally.
With that said, what you said about using LinkedHashMap for declaration implies that maybe we should review the difference between declaration and implementation. Here's a helpful explanation: https://stackoverflow.com/questions/11715485/what-is-the-difference-between-declaration-and-definition-in-java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439685972
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
Review comment:
nit, just put in the try block. It is not used outside.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +78,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
+ _clusterTopologyKeys = new LinkedHashSet<>();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int lastValidTypeIdx = 0;
+ for (int i = 0; i < topologyKeys.length; i++) {
+ if (topologyKeys[i].length() != 0) {
+ _clusterTopologyKeys.add(topologyKeys[i]);
+ lastValidTypeIdx = i;
}
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (_clusterTopologyKeys.size() == 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyKeys[lastValidTypeIdx];
Review comment:
Does "_clusterTopologyKeys.get(_clusterTopologyKeys.size() - 1)" works the same? But you won't need lastValidTypeIdx in that way.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +78,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
+ _clusterTopologyKeys = new LinkedHashSet<>();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int lastValidTypeIdx = 0;
+ for (int i = 0; i < topologyKeys.length; i++) {
+ if (topologyKeys[i].length() != 0) {
+ _clusterTopologyKeys.add(topologyKeys[i]);
+ lastValidTypeIdx = i;
}
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (_clusterTopologyKeys.size() == 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
Review comment:
Where do you update this map? I didn't see in your latest version.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
I talked with Xiaoyuan, since the method logic requires the input map has a fixed iterate order, I suggest her to use the explicit type. Please note that if you input a Map instantiate by HashMap, then the method won't work as expected!
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
Review comment:
It won't be null. Don't overprotect the code, it may hide issues.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String
+ .format("Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ instanceConfig.getDomainAsString(), instanceName), e);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ if (domainAsMap == null || domainAsMap.isEmpty()) {
+ throw new IllegalArgumentException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
+ }
+ int numOfMatchedKeys = 0;
+ for (String key : clusterTopologyKeys) {
+ // if a key does not exist in the instance domain config, using the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value =
+ defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key));
Review comment:
I see you want a lazy initialization. It's good design.
But since you are modifying the input parameter implicitly, it might be clearer to just refer to _defaultDomainPathValues. Otherwise, there is an implicit assumption that the caller needs to pass the same map for all the calls.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String
+ .format("Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ instanceConfig.getDomainAsString(), instanceName), e);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ if (domainAsMap == null || domainAsMap.isEmpty()) {
+ throw new IllegalArgumentException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
+ }
+ int numOfMatchedKeys = 0;
+ for (String key : clusterTopologyKeys) {
+ // if a key does not exist in the instance domain config, using the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value =
+ defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key));
+ } else {
+ numOfMatchedKeys++;
+ }
+ instanceTopologyMap.put(key, value);
+ }
+ if (numOfMatchedKeys != domainAsMap.size()) {
+ logger.warn(
+ "Key-value pairs in InstanceConfig.Domain {} do not align with keys in ClusterConfig.Topology "
+ + "{}, using default domain value instead", instanceConfig.getDomainAsString(),
+ clusterTopologyKeys.toString());
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
Review comment:
Add the corresponding comment as you did for the other 2 conditions above.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String
+ .format("Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ instanceConfig.getDomainAsString(), instanceName), e);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ if (domainAsMap == null || domainAsMap.isEmpty()) {
Review comment:
It won't be null, I guess. Don't overprotect the code, it may hide issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439708594
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +78,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
+ _clusterTopologyKeys = new LinkedHashSet<>();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyKeys = topologyDef.trim().split("/");
+ int lastValidTypeIdx = 0;
+ for (int i = 0; i < topologyKeys.length; i++) {
+ if (topologyKeys[i].length() != 0) {
+ _clusterTopologyKeys.add(topologyKeys[i]);
+ lastValidTypeIdx = i;
}
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (_clusterTopologyKeys.size() == 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
Review comment:
Updated
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String
+ .format("Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ instanceConfig.getDomainAsString(), instanceName), e);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ if (domainAsMap == null || domainAsMap.isEmpty()) {
+ throw new IllegalArgumentException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
+ }
+ int numOfMatchedKeys = 0;
+ for (String key : clusterTopologyKeys) {
+ // if a key does not exist in the instance domain config, using the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length() == 0) {
+ value =
+ defaultDomainPathValuesCache.computeIfAbsent(key, k -> (DEFAULT_DOMAIN_PREFIX + key));
+ } else {
+ numOfMatchedKeys++;
+ }
+ instanceTopologyMap.put(key, value);
+ }
+ if (numOfMatchedKeys != domainAsMap.size()) {
+ logger.warn(
+ "Key-value pairs in InstanceConfig.Domain {} do not align with keys in ClusterConfig.Topology "
+ + "{}, using default domain value instead", instanceConfig.getDomainAsString(),
+ clusterTopologyKeys.toString());
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
Review comment:
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438510939
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +187,129 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ Map<String, String> instanceTopologyMap =
Review comment:
Shouldn't we use interface for declaration (as previous comments suggested)?
Or in this case it's better to explicitly state the ordering is important?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433541650
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
Review comment:
For this in class variable defined here in a private function for class constructor, i guess it is safe to assume the order will be maintained as expected since this private function has the full control of this object. Of course, more detailed comment/doc should be added.
I think it is safe to use the interface map to declare a in class object here. The public util function(computeInstanceTopologyMap) however, takes this LinkedHashMap instance, populate it and return as an out param.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
+ }
}
- pathValueMap.put(type, value);
- }
-
- int weight = insConfig.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
+ } else {
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
}
- return root;
+ return true;
}
-
/**
* Add an end node to the tree, create all the paths to the leaf node if not present.
*/
- private Node addEndNode(Node root, String instanceName, Map<String, String> pathNameMap,
+ private void addEndNode(Node root, String instanceName, LinkedHashMap<String, String> pathNameMap,
Review comment:
That make cense. Will update the pr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439709111
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
Review comment:
Updated.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
+ try {
+ instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys, _defaultDomainPathValues);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys,
+ Map<String, String> defaultDomainPathValuesCache) throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys == null || clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(String
+ .format("Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ instanceConfig.getDomainAsString(), instanceName), e);
}
- String type = values[0];
- String value = values[1];
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ if (domainAsMap == null || domainAsMap.isEmpty()) {
Review comment:
Updated.
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +190,124 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ LinkedHashMap<String, String> instanceTopologyMap;
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443057692
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
Review comment:
I kept the original implementation for performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433002750
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
Review comment:
Let's use `Map` instead...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432892940
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ instanceTopologyMap.put(key, domainAsMap.getOrDefault(key, "Helix_default_" + key));
Review comment:
TFTR. Will update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r442400605
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +191,122 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
- if (zone == null) {
- // we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing ZONE_ID information, fails the rebalance.
- throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
- }
-
+ // TODO: Currently we add disabled instance to the topology tree. Since they are not considered
+ // TODO: in relabalnce, maybe we should skip adding them to the tree for consistence.
+ for (String instanceName : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instanceName);
+ try {
+ LinkedHashMap<String, String> instanceTopologyMap =
+ computeInstanceTopologyMap(_clusterConfig.isTopologyAwareEnabled(), instanceName,
+ insConfig, _clusterTopologyKeys);
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instanceName, instanceTopologyMap, weight, _liveInstances);
+ } catch (IllegalArgumentException e) {
+ if (isInstanceEnabled(_clusterConfig, instanceName, insConfig)) {
+ throw e;
+ } else {
+ logger
+ .warn("Topology setting {} for instance {} is unset or invalid, ignore the instance!",
+ insConfig.getDomainAsString(), instanceName);
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
}
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
}
return root;
}
+ private boolean isInstanceEnabled(ClusterConfig clusterConfig, String instanceName,
+ InstanceConfig instanceConfig) {
+ return (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instanceName)));
+ }
+
/**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
+ * This function returns a LinkedHashMap<String, String> object representing
+ * the topology path for an instance.
+ * LinkedHashMap is used here since the order of the path needs to be preserved
+ * when creating the topology tree.
+ *
+ * @return an LinkedHashMap object representing the topology path for the input instance.
*/
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ private LinkedHashMap<String, String> computeInstanceTopologyMap(boolean isTopologyAwareEnabled,
+ String instanceName, InstanceConfig instanceConfig, LinkedHashSet<String> clusterTopologyKeys)
+ throws IllegalArgumentException {
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (isTopologyAwareEnabled) {
+ if (clusterTopologyKeys.size() == 0) {
+ // Return a ordered map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
+ if (zone == null) {
+ throw new IllegalArgumentException(String
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instanceName));
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
- throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instanceName);
+ } else {
+ /*
+ * Return a ordered map representing the instance path. The topology order is defined in
+ * ClusterConfig.topology.
+ */
+ Map<String, String> domainAsMap = new HashMap<>();
Review comment:
I kept the original logic (before code clean up) here. One difference is that getDomainAsMap wont throw exception if key or value is empty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] zhangmeng916 commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432998959
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
- logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ logger.warn("ZONE_ID for instance {} is not set, ignore the instance!", instance);
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn("Domain for instance {} is not set, ignore the instance!", instance);
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
throw new HelixException(String.format(
- "Domain-Value pair %s for instance %s is not valid, failed the topology-aware placement!",
- pair, ins));
+ "Domain %s for instance %s is not valid, fail the topology-aware placement!",
+ domain, instance));
}
- String type = values[0];
- String value = values[1];
-
- if (!_types.contains(type)) {
- logger.warn(String
- .format("Path %s defined in domain of instance %s not recognized, ignored!", pair,
- ins));
- continue;
+ String[] topologyKeys = topologyDef.trim().split("/");
+ for (String key : topologyKeys) {
+ if (!key.isEmpty()) {
+ // if a key does not exist in the instance domain config, apply the default domain value.
+ String value = domainAsMap.get(key);
+ if (value == null || value.length()==0) {
+ value = _defaultPathPrefix + key;
+ }
+ instanceTopologyMap.put(key, value);
Review comment:
I'd like to know the rationale of setting default value. Basically if the "domain" field is not set, we throw an exception, but if the "domain" is set with a random value, we think it is fine and give it a default zone value. How do we know it doesn't cause problem for rebalance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1043:
URL: https://github.com/apache/helix/pull/1043#issuecomment-647040153
This PR is ready to be merged, approved by @jiajunwang
Code clean up -- Topology.java
This change did come code refactor work for Topology.java. It extracts the sanity check logic in multiple functions into one single function so it can be used else where as well.
Also, mvn test result is updated as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r439644179
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r443092207
##########
File path: helix-core/src/main/java/org/apache/helix/model/InstanceConfig.java
##########
@@ -150,12 +150,22 @@ public String getDomainAsString() {
*/
public Map<String, String> getDomainAsMap() {
String domain = getDomainAsString();
+ Map<String, String> domainAsMap = new HashMap<>();
if (domain == null || domain.isEmpty()) {
- return Collections.emptyMap();
+ return domainAsMap;
+ }
+
+ String[] pathPairs = domain.trim().split(",");
+ for (String pair : pathPairs) {
+ String[] values = pair.trim().split("=");
+ if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Domain-Value pair %s is not valid.", pair));
+ }
+ domainAsMap.put(values[0], values[1]);
Review comment:
Oh I got what you are saying. Yes thats correct. Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
narendly commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r433520419
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +186,106 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
Review comment:
That is acceptable, but what is this order exactly? How should users order the map when passing topology map in?
Would it be better to sort it yourself - would that be easier for the user and be safer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang merged pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
jiajunwang merged pull request #1043:
URL: https://github.com/apache/helix/pull/1043
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r432893951
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -212,123 +185,104 @@ private static Node cloneTree(Node root, Map<Node, Integer> newNodeWeight, Set<N
return newRoot;
}
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithDefaultTopologyDef() {
+ private Node createClusterTree() {
// root
Node root = new Node();
root.setName("root");
root.setId(computeId("root"));
root.setType(Types.ROOT.name());
- for (String ins : _allInstances) {
- InstanceConfig config = _instanceConfigMap.get(ins);
- Map<String, String> pathValueMap = new HashMap<>();
- if (_topologyAwareEnabled) {
- String zone = config.getZoneId();
+ for (String instance : _allInstances) {
+ InstanceConfig insConfig = _instanceConfigMap.get(instance);
+ LinkedHashMap<String, String> instanceTopologyMap = new LinkedHashMap<>();
+ if (computeInstanceTopologyMap(_clusterConfig, instance, insConfig, instanceTopologyMap)) {
+ int weight = insConfig.getWeight();
+ if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
+ weight = DEFAULT_NODE_WEIGHT;
+ }
+ addEndNode(root, instance, instanceTopologyMap, weight, _liveInstances);
+ }
+ }
+ return root;
+ }
+
+ private boolean computeInstanceTopologyMap(ClusterConfig clusterConfig, String instance,
+ InstanceConfig instanceConfig, LinkedHashMap<String, String> instanceTopologyMap) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
+ String topologyDef = clusterConfig.getTopology();
+ if (topologyDef == null) {
+ // Return a map using default cluster topology definition, i,e. /root/zone/instance
+ String zone = instanceConfig.getZoneId();
if (zone == null) {
// we have the hierarchy style of domain id for instance.
- if (config.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
// if enabled instance missing ZONE_ID information, fails the rebalance.
throw new HelixException(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
+ .format("ZONE_ID for instance %s is not set, fail the topology-aware placement!",
+ instance));
} else {
// if the disabled instance missing ZONE setting, ignore it should be fine.
logger.warn(String
- .format("ZONE_ID for instance %s is not set, failed the topology-aware placement!",
- ins));
- continue;
+ .format("ZONE_ID for instance %s is not set, ignore the instance!", instance));
+ return false;
}
-
}
- pathValueMap.put(Types.ZONE.name(), zone);
- }
- pathValueMap.put(Types.INSTANCE.name(), ins);
- int weight = config.getWeight();
- if (weight < 0 || weight == InstanceConfig.WEIGHT_NOT_SET) {
- weight = DEFAULT_NODE_WEIGHT;
- }
- root = addEndNode(root, ins, pathValueMap, weight, _liveInstances);
- }
- return root;
- }
-
- /**
- * Creates a tree representing the cluster structure using default cluster topology definition
- * (i,e no topology definition given and no domain id set).
- */
- private Node createClusterTreeWithCustomizedTopology() {
- // root
- Node root = new Node();
- root.setName("root");
- root.setId(computeId("root"));
- root.setType(Types.ROOT.name());
-
- for (String ins : _allInstances) {
- InstanceConfig insConfig = _instanceConfigMap.get(ins);
- String domain = insConfig.getDomainAsString();
- if (domain == null) {
- if (insConfig.getInstanceEnabled() && (_clusterConfig.getDisabledInstances() == null
- || !_clusterConfig.getDisabledInstances().containsKey(ins))) {
- // if enabled instance missing domain information, fails the rebalance.
- throw new HelixException(String
- .format("Domain for instance %s is not set, failed the topology-aware placement!",
- ins));
- } else {
- // if the disabled instance missing domain setting, ignore it should be fine.
- logger
- .warn(String.format("Domain for instance %s is not set, ignore the instance!", ins));
- continue;
+ instanceTopologyMap.put(Types.ZONE.name(), zone);
+ instanceTopologyMap.put(Types.INSTANCE.name(), instance);
+ } else {
+ /*
+ * Return a map representing the cluster structure using cluster topology defined in
+ * TOPOLOGY in ClusterConfig.
+ */
+ String domain = instanceConfig.getDomainAsString();
+ if (domain == null || domain.isEmpty()) {
+ if (instanceConfig.getInstanceEnabled() && (clusterConfig.getDisabledInstances() == null
+ || !clusterConfig.getDisabledInstances().containsKey(instance))) {
+ // if enabled instance missing domain information, fails the rebalance.
+ throw new HelixException(String
+ .format("Domain for instance %s is not set, fail the topology-aware placement!",
+ instance));
+ } else {
+ // if the disabled instance missing domain setting, ignore it should be fine.
+ logger.warn(
+ String.format("Domain for instance %s is not set, ignore the instance!", instance));
+ return false;
+ }
}
- }
-
- String[] pathPairs = domain.trim().split(",");
- Map<String, String> pathValueMap = new HashMap<>();
- for (String pair : pathPairs) {
- String[] values = pair.trim().split("=");
- if (values.length != 2 || values[0].isEmpty() || values[1].isEmpty()) {
+ Map<String, String> domainAsMap;
+ try {
+ domainAsMap = instanceConfig.getDomainAsMap();
+ } catch (IllegalArgumentException e) {
Review comment:
GetDomainAsMap will throw IllegalArgumentException if
1. The Splitter.on() param matches the empty string.
2. Has duplicate Key.
For example the controller does not know how to categorize the instance if user specified two different rack.
There is one behavior change (witch I should have documented in the pr description) is that, GetDomainAsMap won't throw exception if either key or value is an empty string. I think the new behavior is reasonable since
1. We ignore keys in DOMAIN that are not defined in ClusterConfig. When we have empty key, they should be treated as non-defined key. Instead of throw exception, it is ok to just ignore them. They are no harm to the topology tree anyway.
2. We will use default values for keys that are defined in ClusterConfig but missing InstanceConfig.DOMAIN. If there is an empty value defined, we should use the default value as well.
There will be a warning log if the keys mismatch with each other.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu commented on a change in pull request #1043: Code clean up Topology.java
Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1043:
URL: https://github.com/apache/helix/pull/1043#discussion_r438505561
##########
File path: helix-core/src/main/java/org/apache/helix/controller/rebalancer/topology/Topology.java
##########
@@ -81,62 +75,43 @@ public Topology(final List<String> allNodes, final List<String> liveNodes,
throw new HelixException(String.format("Config for instances %s is not found!",
_allInstances.removeAll(_instanceConfigMap.keySet())));
}
-
_clusterConfig = clusterConfig;
- _types = new LinkedHashSet<>();
- _topologyAwareEnabled = clusterConfig.isTopologyAwareEnabled();
- if (_topologyAwareEnabled) {
+ if (clusterConfig.isTopologyAwareEnabled()) {
String topologyDef = _clusterConfig.getTopology();
if (topologyDef != null) {
// Customized cluster topology definition is configured.
- String[] types = topologyDef.trim().split("/");
- for (int i = 0; i < types.length; i++) {
- if (types[i].length() != 0) {
- _types.add(types[i]);
+ String[] topologyStr = topologyDef.trim().split("/");
+ int lastValidTypeIdx = topologyStr.length - 1;
+ while(lastValidTypeIdx >= 0) {
+ if (topologyStr[lastValidTypeIdx].length() != 0) {
+ break;
}
+ --lastValidTypeIdx;
}
- if (_types.size() == 0) {
- logger.error("Invalid cluster topology definition " + topologyDef);
+ if (lastValidTypeIdx < 0) {
throw new HelixException("Invalid cluster topology definition " + topologyDef);
- } else {
- String lastType = null;
- for (String type : _types) {
- _defaultDomainPathValues.put(type, "Helix_default_" + type);
- lastType = type;
- }
- _endNodeType = lastType;
- _faultZoneType = clusterConfig.getFaultZoneType();
- if (_faultZoneType == null) {
- _faultZoneType = _endNodeType;
- }
- if (!_types.contains(_faultZoneType)) {
- throw new HelixException(String
- .format("Invalid fault zone type %s, not present in topology definition %s.",
- _faultZoneType, topologyDef));
- }
- _useDefaultTopologyDef = false;
+ }
+ _endNodeType = topologyStr[lastValidTypeIdx];
+ _faultZoneType = clusterConfig.getFaultZoneType();
+ if (_faultZoneType == null) {
+ _faultZoneType = _endNodeType;
+ }
+ if (Arrays.stream(topologyStr).noneMatch(type -> type.equals(_faultZoneType))) {
Review comment:
TFTR. Please correct me if I am wrong.
Comparing to previous approach (set. contains()), the stream.noneMatch() does introduced a o(n)worst case complexity.
However, since there is no new LinkedHashSet created, it saved a O(n) time for constructing the LinkedHashSet and a O(n) space complexity for the object itself.
It is better to pass the already parsed array to computeInstanceTopologyMap so we do not compute again in line 274. However, I think we could just use the original array topologyStr, no need to introduce another LinkedHashSet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org