You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "MarkGaox (via GitHub)" <gi...@apache.org> on 2025/01/03 22:49:05 UTC
[PR] Markgaox/support zone based virtual topology [helix]
MarkGaox opened a new pull request, #2986:
URL: https://github.com/apache/helix/pull/2986
### Issues
- [X] My PR addresses the following Helix issues and references them in the PR description:
#2985
(#200 - Link your issue number here: You can write "Fixes #XXX". Please use the proper keyword so that the issue gets closed automatically. See https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue
Any of the following keywords can be used: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved)
### Description
- [X] Here are some details about my PR, including screenshots of any UI changes:
This PR support zone based virtual topology assignment algorithm. In the rest service, introduced a configurable parameter enabling users to select the desired virtual topology assignment algorithm (e.g.,instance_based or zone_based). Additionally, implemented a zone-based assignment algorithm that directly maps physical zones to virtual zones.
### Tests
- [X] The following tests are written for this issue:
```
mvn test -Dtest=TestVirtualTopologyGroupService,TestFaultZoneBasedVirtualGroupAssignment,TestClusterService,TestVirtualTopologyGroupAssignment,TestClusterAccessor -pl helix-rest
```
### Changes that Break Backward Compatibility (Optional)
- This PR doesn't have backward compatibility issue.
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on PR #2986:
URL: https://github.com/apache/helix/pull/2986#issuecomment-2611206238
Generally LGTM. nit comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1927799795
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/FaultZoneBasedVirtualGroupAssignmentAlgorithm.java:
##########
@@ -59,137 +59,127 @@ public static FaultZoneBasedVirtualGroupAssignmentAlgorithm getInstance() {
@Override
public Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
- Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping) {
- // Build instance-to-zone mapping for quick zone lookups.
- Map<String, String> instanceToZoneMapping = new HashMap<>();
- for (Map.Entry<String, Set<String>> entry : zoneMapping.entrySet()) {
- for (String instance : entry.getValue()) {
- instanceToZoneMapping.put(instance, entry.getKey());
- }
- }
- // Collect all instances from zoneMapping
- Set<String> allInstances = zoneMapping.values()
- .stream()
- .flatMap(Set::stream)
- .collect(Collectors.toSet());
-
+ Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualGroupToInstancesMap) {
// 1. If the number of requested virtual groups differs from the current assignment size,
// we must do a fresh assignment (the existing distribution is invalid).
- if (numGroups != virtualZoneMapping.size()) {
+ if (numGroups != virtualGroupToInstancesMap.size()) {
Map<String, Set<String>> newAssignment = new HashMap<>();
for (int i = 0; i < numGroups; i++) {
newAssignment.put(computeVirtualGroupId(i, virtualGroupName), new HashSet<>());
}
- // Assign all instances from scratch in a balanced manner.
- return distributeUnassignedZones(newAssignment, allInstances, instanceToZoneMapping);
+ // Assign all zones from scratch in a balanced manner.
Review Comment:
nit: add comment for step 2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1904668523
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/FaultZoneBasedVirtualGroupAssignmentAlgorithm.java:
##########
@@ -0,0 +1,73 @@
+package org.apache.helix.cloud.topology;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
+
+import static org.apache.helix.util.VirtualTopologyUtil.computeVirtualGroupId;
+
+/**
+ * A strategy that densely assign virtual groups with input instance list, it doesn't move to the next one until
+ * the current one is filled.
+ * Given that instances.size = instancesPerGroup * numGroups + residuals,
+ * we break [residuals] into the first few groups, as a result each virtual group will have
+ * either [instancesPerGroup] or [instancesPerGroup + 1] instances.
+ */
+public class FaultZoneBasedVirtualGroupAssignmentAlgorithm implements VirtualGroupAssignmentAlgorithm {
+
+ private static final FaultZoneBasedVirtualGroupAssignmentAlgorithm _instance = new FaultZoneBasedVirtualGroupAssignmentAlgorithm();
+
+ private FaultZoneBasedVirtualGroupAssignmentAlgorithm() { }
+
+ public static FaultZoneBasedVirtualGroupAssignmentAlgorithm getInstance() {
+ return _instance;
+ }
+
+ @Override
Review Comment:
signature
Would be great if provide an example
##########
helix-core/src/main/java/org/apache/helix/cloud/constants/VirtualTopologyGroupConstants.java:
##########
@@ -24,7 +24,12 @@ public class VirtualTopologyGroupConstants {
public static final String GROUP_NAME = "virtualTopologyGroupName";
public static final String GROUP_NUMBER = "virtualTopologyGroupNumber";
public static final String AUTO_MAINTENANCE_MODE_DISABLED = "autoMaintenanceModeDisabled";
+ public static final String ASSIGNMENT_ALGORITHM = "virtualTopologyAssignmentAlgorithm";
Review Comment:
maybe ASSIGNMENT_ALGORITHM_TYPE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1924494364
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/FaultZoneBasedVirtualGroupAssignmentAlgorithm.java:
##########
@@ -0,0 +1,195 @@
+package org.apache.helix.cloud.topology;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.stream.Collectors;
+
+import org.apache.commons.math3.util.Pair;
+
+import static org.apache.helix.util.VirtualTopologyUtil.computeVirtualGroupId;
+
+/**
+ * A virtual group assignment algorithm that assigns zones and their instances to virtual groups
+ * a way that preserves existing zone-to-group assignments whenever possible, and balances any
+ * remaining unassigned zones across the least-loaded groups. If the requested number of groups
+ * differs from the existing assignment, a new distribution is computed. Otherwise, if a zone
+ * already exists in the provided assignment, all its instances (including newly discovered ones)
+ * are placed in the same group, ensuring no zone is split across multiple virtual groups.
+ */
+public class FaultZoneBasedVirtualGroupAssignmentAlgorithm implements VirtualGroupAssignmentAlgorithm {
+
+ private static final FaultZoneBasedVirtualGroupAssignmentAlgorithm _instance =
+ new FaultZoneBasedVirtualGroupAssignmentAlgorithm();
+
+ private FaultZoneBasedVirtualGroupAssignmentAlgorithm() {
+ }
+
+ public static FaultZoneBasedVirtualGroupAssignmentAlgorithm getInstance() {
+ return _instance;
+ }
+
+ @Override
+ public Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
+ Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping) {
Review Comment:
nit: also please align naming, we have Physical zone and Virtual group I think. Virtual zone is confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1924480874
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/VirtualGroupAssignmentAlgorithm.java:
##########
@@ -34,5 +34,5 @@ public interface VirtualGroupAssignmentAlgorithm {
* @return the assignment as mapping from virtual group ID to instanceIds
*/
Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
Review Comment:
Update comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1924494000
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/FaultZoneBasedVirtualGroupAssignmentAlgorithm.java:
##########
@@ -0,0 +1,195 @@
+package org.apache.helix.cloud.topology;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.stream.Collectors;
+
+import org.apache.commons.math3.util.Pair;
+
+import static org.apache.helix.util.VirtualTopologyUtil.computeVirtualGroupId;
+
+/**
+ * A virtual group assignment algorithm that assigns zones and their instances to virtual groups
+ * a way that preserves existing zone-to-group assignments whenever possible, and balances any
+ * remaining unassigned zones across the least-loaded groups. If the requested number of groups
+ * differs from the existing assignment, a new distribution is computed. Otherwise, if a zone
+ * already exists in the provided assignment, all its instances (including newly discovered ones)
+ * are placed in the same group, ensuring no zone is split across multiple virtual groups.
+ */
+public class FaultZoneBasedVirtualGroupAssignmentAlgorithm implements VirtualGroupAssignmentAlgorithm {
+
+ private static final FaultZoneBasedVirtualGroupAssignmentAlgorithm _instance =
+ new FaultZoneBasedVirtualGroupAssignmentAlgorithm();
+
+ private FaultZoneBasedVirtualGroupAssignmentAlgorithm() {
+ }
+
+ public static FaultZoneBasedVirtualGroupAssignmentAlgorithm getInstance() {
+ return _instance;
+ }
+
+ @Override
+ public Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
+ Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping) {
+ // Build instance-to-zone mapping for quick zone lookups.
+ Map<String, String> instanceToZoneMapping = new HashMap<>();
Review Comment:
I think this part of the algo can be simplified. Please correct me if I am wrong.
Input of this function:
Current Zone->instance mapping. This is includes all instances and their zone info. Some(or all) instances are not included in previous assignment
Current VG -> instance mapping.
Return value:
new VG -> instance mapping.
Now we compute so many in between data structure, we build map and reverse map for same set of information. I think it can be improved to:
2. If current VG -> instance mapping is not null, we compute VG->Zone mapping of existing assignment.
3. Create new assignment result map, for each VG, dd all instances in those zones.
4. Assign remaining Zones (these are now zones that are not existed in previous mapping)
a. We build map of zone -> instance count.
b. Sort the VG in to min heap, Dont need to build a map, but sort based on instance size (we have the VG to zones map and zone to number of instance map)
c. Sort remaining zones in decending order using zone -> instance count. Again, we dont need to build a map.
d. Do the assignment, update map.
In this way, we only build 1 data structure in between, and algo is simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1924494000
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/FaultZoneBasedVirtualGroupAssignmentAlgorithm.java:
##########
@@ -0,0 +1,195 @@
+package org.apache.helix.cloud.topology;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.stream.Collectors;
+
+import org.apache.commons.math3.util.Pair;
+
+import static org.apache.helix.util.VirtualTopologyUtil.computeVirtualGroupId;
+
+/**
+ * A virtual group assignment algorithm that assigns zones and their instances to virtual groups
+ * a way that preserves existing zone-to-group assignments whenever possible, and balances any
+ * remaining unassigned zones across the least-loaded groups. If the requested number of groups
+ * differs from the existing assignment, a new distribution is computed. Otherwise, if a zone
+ * already exists in the provided assignment, all its instances (including newly discovered ones)
+ * are placed in the same group, ensuring no zone is split across multiple virtual groups.
+ */
+public class FaultZoneBasedVirtualGroupAssignmentAlgorithm implements VirtualGroupAssignmentAlgorithm {
+
+ private static final FaultZoneBasedVirtualGroupAssignmentAlgorithm _instance =
+ new FaultZoneBasedVirtualGroupAssignmentAlgorithm();
+
+ private FaultZoneBasedVirtualGroupAssignmentAlgorithm() {
+ }
+
+ public static FaultZoneBasedVirtualGroupAssignmentAlgorithm getInstance() {
+ return _instance;
+ }
+
+ @Override
+ public Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
+ Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping) {
+ // Build instance-to-zone mapping for quick zone lookups.
+ Map<String, String> instanceToZoneMapping = new HashMap<>();
Review Comment:
I think this part of the algo can be simplified. Please correct me if I am wrong.
Input of this function:
Current Zone->instance mapping. This is includes all instances and their zone info. Some(or all) instances are not included in previous assignment
Current VG -> instance mapping.
Return value:
new VG -> instance mapping.
Now we compute so many in between data structure, we build map and reverse map for same set of information. I think it can be improved to:
2. If current VG -> instance mapping is not null, we compute VG->Zone mapping of existing assignment.
3. Create new assignment result map, for each VG, dd all instances in those zones.
4. Assign remaining Zones (these are now zones that are not existed in previous mapping)
a. We build map of zone -> instance count.
b. Sort the VG in to min heap, Dont need to build a map, but sort based on instance size (we have the VG to zones map and zone to number of instance map)
c. Sort remaining zones in decending order using zone -> instance count. Again, we dont need to build a map.
d. Do the assignment, update map.
In this way, we only build 1 data structure in between, and algo is simpler.
-
- 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1924255760
##########
helix-rest/src/main/java/org/apache/helix/rest/server/service/VirtualTopologyGroupService.java:
##########
@@ -60,15 +62,15 @@ public class VirtualTopologyGroupService {
private final ClusterService _clusterService;
private final ConfigAccessor _configAccessor;
private final HelixDataAccessor _dataAccessor;
- private final VirtualGroupAssignmentAlgorithm _assignmentAlgorithm;
+ private VirtualGroupAssignmentAlgorithm _assignmentAlgorithm;
public VirtualTopologyGroupService(HelixAdmin helixAdmin, ClusterService clusterService,
ConfigAccessor configAccessor, HelixDataAccessor dataAccessor) {
_helixAdmin = helixAdmin;
_clusterService = clusterService;
_configAccessor = configAccessor;
_dataAccessor = dataAccessor;
- _assignmentAlgorithm = FifoVirtualGroupAssignmentAlgorithm.getInstance();
+ _assignmentAlgorithm = FaultZoneBasedVirtualGroupAssignmentAlgorithm.getInstance(); // default assignment algorithm
Review Comment:
We should not change the algorithm default or this will break backwards compatibility.
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/VirtualGroupAssignmentAlgorithm.java:
##########
@@ -34,5 +34,5 @@ public interface VirtualGroupAssignmentAlgorithm {
* @return the assignment as mapping from virtual group ID to instanceIds
*/
Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
- Map<String, Set<String>> zoneMapping);
+ Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping);
Review Comment:
Let's add a new signature with default implementation:
Ex:
```
default Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping) {
omputeAssignment(int numGroups, String virtualGroupName,
Map<String, Set<String>> zoneMapping);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue merged PR #2986:
URL: https://github.com/apache/helix/pull/2986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1927802152
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/FaultZoneBasedVirtualGroupAssignmentAlgorithm.java:
##########
@@ -59,137 +59,127 @@ public static FaultZoneBasedVirtualGroupAssignmentAlgorithm getInstance() {
@Override
public Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
- Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping) {
- // Build instance-to-zone mapping for quick zone lookups.
- Map<String, String> instanceToZoneMapping = new HashMap<>();
- for (Map.Entry<String, Set<String>> entry : zoneMapping.entrySet()) {
- for (String instance : entry.getValue()) {
- instanceToZoneMapping.put(instance, entry.getKey());
- }
- }
- // Collect all instances from zoneMapping
- Set<String> allInstances = zoneMapping.values()
- .stream()
- .flatMap(Set::stream)
- .collect(Collectors.toSet());
-
+ Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualGroupToInstancesMap) {
// 1. If the number of requested virtual groups differs from the current assignment size,
// we must do a fresh assignment (the existing distribution is invalid).
- if (numGroups != virtualZoneMapping.size()) {
+ if (numGroups != virtualGroupToInstancesMap.size()) {
Map<String, Set<String>> newAssignment = new HashMap<>();
for (int i = 0; i < numGroups; i++) {
newAssignment.put(computeVirtualGroupId(i, virtualGroupName), new HashSet<>());
}
- // Assign all instances from scratch in a balanced manner.
- return distributeUnassignedZones(newAssignment, allInstances, instanceToZoneMapping);
+ // Assign all zones from scratch in a balanced manner.
+ distributeUnassignedZones(newAssignment, new ArrayList<>(zoneMapping.keySet()), zoneMapping);
+ return constructResult(newAssignment, zoneMapping);
}
- // Build a deep copy of the current assignment to avoid mutating it directly.
- Map<String, Set<String>> updatedAssignment = deepCopy(virtualZoneMapping);
-
- // 2. Identify unassigned instances.
- // Based on the existing assignment, build zone -> virtual group mapping for quick reference.
- Map<String, String> zoneToVirtualGroupMapping = new HashMap<>();
- Set<String> assignedInstances = new HashSet<>();
- for (Map.Entry<String, Set<String>> entry : updatedAssignment.entrySet()) {
+ // Build instance-to-zone mapping for quick zone lookups.
+ Map<String, String> instanceToZoneMapping = new HashMap<>();
+ for (Map.Entry<String, Set<String>> entry : zoneMapping.entrySet()) {
for (String instance : entry.getValue()) {
- String zone = instanceToZoneMapping.get(instance);
- assignedInstances.add(instance);
- zoneToVirtualGroupMapping.put(zone, entry.getKey());
+ instanceToZoneMapping.put(instance, entry.getKey());
}
}
- Set<String> unassigned = new HashSet<>(allInstances);
- unassigned.removeAll(assignedInstances);
-
- // 3. Attempt to place unassigned instances in existing virtual groups if their zones are
- // already present in the assignment.
- Iterator<String> iterator = unassigned.iterator();
- while (iterator.hasNext()) {
- String instance = iterator.next();
- String zone = instanceToZoneMapping.get(instance);
- if (zoneToVirtualGroupMapping.containsKey(zone)) {
- updatedAssignment.get(zoneToVirtualGroupMapping.get(zone)).add(instance);
- iterator.remove();
+
+ // Copy zoneMapping for tracking which zones are unassigned.
+ Map<String, Set<String>> unassignedZoneToInstances = copyZoneMapping(zoneMapping);
+
+ // Build virtual group -> zone mapping and remove assigned zones from the unassigned list
+ Map<String, Set<String>> virtualGroupToZoneMapping = new HashMap<>();
+ for (Map.Entry<String, Set<String>> entry : virtualGroupToInstancesMap.entrySet()) {
+ virtualGroupToZoneMapping.putIfAbsent(entry.getKey(), new HashSet<>());
+ for (String instance : entry.getValue()) {
+ String zone = instanceToZoneMapping.get(instance);
+ virtualGroupToZoneMapping.get(entry.getKey()).add(zone);
+ unassignedZoneToInstances.remove(zone);
Review Comment:
you can do early exit here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2986:
URL: https://github.com/apache/helix/pull/2986#discussion_r1927800065
##########
helix-core/src/main/java/org/apache/helix/cloud/topology/FaultZoneBasedVirtualGroupAssignmentAlgorithm.java:
##########
@@ -59,137 +59,127 @@ public static FaultZoneBasedVirtualGroupAssignmentAlgorithm getInstance() {
@Override
public Map<String, Set<String>> computeAssignment(int numGroups, String virtualGroupName,
- Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualZoneMapping) {
- // Build instance-to-zone mapping for quick zone lookups.
- Map<String, String> instanceToZoneMapping = new HashMap<>();
- for (Map.Entry<String, Set<String>> entry : zoneMapping.entrySet()) {
- for (String instance : entry.getValue()) {
- instanceToZoneMapping.put(instance, entry.getKey());
- }
- }
- // Collect all instances from zoneMapping
- Set<String> allInstances = zoneMapping.values()
- .stream()
- .flatMap(Set::stream)
- .collect(Collectors.toSet());
-
+ Map<String, Set<String>> zoneMapping, Map<String, Set<String>> virtualGroupToInstancesMap) {
// 1. If the number of requested virtual groups differs from the current assignment size,
// we must do a fresh assignment (the existing distribution is invalid).
- if (numGroups != virtualZoneMapping.size()) {
+ if (numGroups != virtualGroupToInstancesMap.size()) {
Map<String, Set<String>> newAssignment = new HashMap<>();
for (int i = 0; i < numGroups; i++) {
newAssignment.put(computeVirtualGroupId(i, virtualGroupName), new HashSet<>());
}
- // Assign all instances from scratch in a balanced manner.
- return distributeUnassignedZones(newAssignment, allInstances, instanceToZoneMapping);
+ // Assign all zones from scratch in a balanced manner.
+ distributeUnassignedZones(newAssignment, new ArrayList<>(zoneMapping.keySet()), zoneMapping);
+ return constructResult(newAssignment, zoneMapping);
}
Review Comment:
nit: add comment for step 2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Support zone based virtual topology assignment algorithm [helix]
Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on PR #2986:
URL: https://github.com/apache/helix/pull/2986#issuecomment-2623670736
Thanks @xyuanlu for reviewing this PR. It's ready to merge.
Final commit message: Support zone based virtual topology assignment algorithm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org