You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "himanshukandwal (via GitHub)" <gi...@apache.org> on 2023/10/03 18:04:44 UTC
[PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
himanshukandwal opened a new pull request, #2639:
URL: https://github.com/apache/helix/pull/2639
### Issues
- [x] My PR addresses the following Helix issues and references them in the PR description:
#2638
### Description
- [x] Here are some details about my PR, including screenshots of any UI changes:
Where the Hard Constraint fails, the error message is very generic and it doesn't give much information on which capacity dimension failed and with what values.
In this PR, we instead of returning a boolean result, we are returning ValidationResult class object that contains whether the validation has passed or not, and if not it will also contain an additional error-message.
### Tests
- [x] The following tests are written for this issue:
Improved Following Tests classes:
* TestConstraintBasedAlgorithm
* TestFaultZoneAwareConstraint
* TestNodeCapacityConstraint
* TestNodeMaxPartitionLimitConstraint
* TestPartitionActivateConstraint
* TestReplicaActivateConstraint
* TestSamePartitionOnInstanceConstraint
* TestValidGroupTagConstraint
- The following is the result of the "mvn test" command on the appropriate module:
``` console
END testMsgConstraint() at Tue Oct 03 10:59:40 PDT 2023
START testStateConstraint() at Tue Oct 03 10:59:40 PDT 2023
{RESOURCE=TestDB, STATE=MASTER} matches(3): [{STATE=MASTER}:1, {STATE=MASTER, RESOURCE=TestDB}:1, {STATE=MASTER, RESOURCE=.*}:2]
{RESOURCE=MyDB, STATE=MASTER} matches(2): [{STATE=MASTER}:1, {STATE=MASTER, RESOURCE=.*}:2]
END testStateConstraint() at Tue Oct 03 10:59:40 PDT 2023
AfterClass: TestConstraint called.
Shut down zookeeper at port 2183 in thread main
[INFO] Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.234 s - in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 32, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- jacoco:0.8.6:report (generate-code-coverage-report) @ helix-core ---
[INFO] Loading execution data file /Users/hkandwal/Documents/workspaces/projects/helix_os_hk/helix-core/target/jacoco.exec
[INFO] Analyzed bundle 'Apache Helix :: Core' with 949 classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.203 s
[INFO] Finished at: 2023-10-03T10:59:42-07:00
[INFO] ------------------------------------------------------------------------
```
### Changes that Break Backward Compatibility (Optional)
- My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
(Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
### Documentation (Optional)
- In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2639:
URL: https://github.com/apache/helix/pull/2639#discussion_r1344684197
##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/ConstraintBasedAlgorithm.java:
##########
@@ -120,24 +120,24 @@ public OptimalAssignment calculate(ClusterModel clusterModel) throws HelixRebala
private Optional<AssignableNode> getNodeWithHighestPoints(AssignableReplica replica,
List<AssignableNode> assignableNodes, ClusterContext clusterContext,
Set<String> busyInstances, OptimalAssignment optimalAssignment) {
- Map<AssignableNode, List<HardConstraint>> hardConstraintFailures = new ConcurrentHashMap<>();
+ Map<AssignableNode, List<String>> hardConstraintFailures = new ConcurrentHashMap<>();
List<AssignableNode> candidateNodes = assignableNodes.parallelStream().filter(candidateNode -> {
boolean isValid = true;
// need to record all the failure reasons and it gives us the ability to debug/fix the runtime
// cluster environment
for (HardConstraint hardConstraint : _hardConstraints) {
- if (!hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext)) {
+ ValidationResult validationResult = hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext);
Review Comment:
Got it. I will then overload this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2639:
URL: https://github.com/apache/helix/pull/2639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2639:
URL: https://github.com/apache/helix/pull/2639#discussion_r1344730248
##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/ConstraintBasedAlgorithm.java:
##########
@@ -120,24 +120,24 @@ public OptimalAssignment calculate(ClusterModel clusterModel) throws HelixRebala
private Optional<AssignableNode> getNodeWithHighestPoints(AssignableReplica replica,
List<AssignableNode> assignableNodes, ClusterContext clusterContext,
Set<String> busyInstances, OptimalAssignment optimalAssignment) {
- Map<AssignableNode, List<HardConstraint>> hardConstraintFailures = new ConcurrentHashMap<>();
+ Map<AssignableNode, List<String>> hardConstraintFailures = new ConcurrentHashMap<>();
List<AssignableNode> candidateNodes = assignableNodes.parallelStream().filter(candidateNode -> {
boolean isValid = true;
// need to record all the failure reasons and it gives us the ability to debug/fix the runtime
// cluster environment
for (HardConstraint hardConstraint : _hardConstraints) {
- if (!hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext)) {
+ ValidationResult validationResult = hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext);
Review Comment:
thanks for confirming it. If i have to summarize your change, it is:
- change the method signature to return ValidationResult. If pass, great but if it failed, it will produce result.
- For each placement, today we do generate why we couldn't place something (even though in generic term, we still get the result), so how is this going to help the customer?
Can we clarify with our customer if this is what we will provide?
```
2023/07/30 00:56:51.769 ERROR [PartialRebalanceRunner] [pool-28-thread-1] [helix] [] Failed to calculate best possible assignment!
org.apache.helix.HelixRebalanceException: Unable to find any available candidate node for partition HireIdentity_53;
Fail reasons: {HireIdentity-HireIdentity_53-SLAVE=i
{ ltx1-app64342.prod.linkedin.com_11932=[Node has insufficient capacity],
ltx1-app64356.prod.linkedin.com_11932=[Node has insufficient capacity],
ltx1-app49575.prod.linkedin.com_11932=[A fault zone cannot contain more than 1 replica of same partition, Node has insufficient capacity],
ltx1-app81203.prod.linkedin.com_11932=[A fault zone cannot contain more than 1 replica of same partition, Node has insufficient capacity],
ltx1-app81873.prod.linkedin.com_11932=[A fault zone cannot contain more than 1 replica of same partition, Node has insufficient capacity],
ltx1-app64336.prod.linkedin.com_11932=[Node has insufficient capacity],
ltx1-app82563.prod.linkedin.com_11932=[Node has insufficient capacity],
ltx1-app49842.prod.linkedin.com_11932=[A fault zone cannot contain more than 1 replica of same partition, Node has insufficient capacity],
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2639:
URL: https://github.com/apache/helix/pull/2639#discussion_r1358956177
##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/HardConstraint.java:
##########
@@ -33,15 +34,46 @@ abstract class HardConstraint {
* Check if the replica could be assigned to the node
* @return True if the proposed assignment is valid; False otherwise
*/
- abstract boolean isAssignmentValid(AssignableNode node, AssignableReplica replica,
+ abstract ValidationResult isAssignmentValid(AssignableNode node, AssignableReplica replica,
ClusterContext clusterContext);
+
/**
- * Return class name by default as description if it's explanatory enough, child class could override
- * the method and add more detailed descriptions
- * @return The detailed description of hard constraint
+ * Stores the assignment validation result and the error message, in case of failure.
*/
- String getDescription() {
- return getClass().getName();
+ public static class ValidationResult {
Review Comment:
Why we need to have this kind of result? I am not fully convinced to have this structure. If a failure of a message require this new object. It will generated a lot of object during pipelines can cause GC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2639:
URL: https://github.com/apache/helix/pull/2639#discussion_r1363183180
##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/HardConstraint.java:
##########
@@ -33,15 +34,46 @@ abstract class HardConstraint {
* Check if the replica could be assigned to the node
* @return True if the proposed assignment is valid; False otherwise
*/
- abstract boolean isAssignmentValid(AssignableNode node, AssignableReplica replica,
+ abstract ValidationResult isAssignmentValid(AssignableNode node, AssignableReplica replica,
ClusterContext clusterContext);
+
/**
- * Return class name by default as description if it's explanatory enough, child class could override
- * the method and add more detailed descriptions
- * @return The detailed description of hard constraint
+ * Stores the assignment validation result and the error message, in case of failure.
*/
- String getDescription() {
- return getClass().getName();
+ public static class ValidationResult {
Review Comment:
Yes, discussed offline. Changing the logic to just log the failed constraint (debug mode) and keeping the rest of the code as it is. Thanks Junkai!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2639:
URL: https://github.com/apache/helix/pull/2639#discussion_r1344697416
##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/ConstraintBasedAlgorithm.java:
##########
@@ -120,24 +120,24 @@ public OptimalAssignment calculate(ClusterModel clusterModel) throws HelixRebala
private Optional<AssignableNode> getNodeWithHighestPoints(AssignableReplica replica,
List<AssignableNode> assignableNodes, ClusterContext clusterContext,
Set<String> busyInstances, OptimalAssignment optimalAssignment) {
- Map<AssignableNode, List<HardConstraint>> hardConstraintFailures = new ConcurrentHashMap<>();
+ Map<AssignableNode, List<String>> hardConstraintFailures = new ConcurrentHashMap<>();
List<AssignableNode> candidateNodes = assignableNodes.parallelStream().filter(candidateNode -> {
boolean isValid = true;
// need to record all the failure reasons and it gives us the ability to debug/fix the runtime
// cluster environment
for (HardConstraint hardConstraint : _hardConstraints) {
- if (!hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext)) {
+ ValidationResult validationResult = hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext);
Review Comment:
Actually if we see, `HardConstraint` is not a public class and is package local. So it is not accessible and extensible. We should be good with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2639:
URL: https://github.com/apache/helix/pull/2639#discussion_r1344746289
##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/ConstraintBasedAlgorithm.java:
##########
@@ -120,24 +120,24 @@ public OptimalAssignment calculate(ClusterModel clusterModel) throws HelixRebala
private Optional<AssignableNode> getNodeWithHighestPoints(AssignableReplica replica,
List<AssignableNode> assignableNodes, ClusterContext clusterContext,
Set<String> busyInstances, OptimalAssignment optimalAssignment) {
- Map<AssignableNode, List<HardConstraint>> hardConstraintFailures = new ConcurrentHashMap<>();
+ Map<AssignableNode, List<String>> hardConstraintFailures = new ConcurrentHashMap<>();
List<AssignableNode> candidateNodes = assignableNodes.parallelStream().filter(candidateNode -> {
boolean isValid = true;
// need to record all the failure reasons and it gives us the ability to debug/fix the runtime
// cluster environment
for (HardConstraint hardConstraint : _hardConstraints) {
- if (!hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext)) {
+ ValidationResult validationResult = hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext);
Review Comment:
Currently we produce a static response for a type of Constraint Failure, which does not capture the details of the failing constraint, as say for Capacity there are many dimensions: disk, partcount, cu ..
With this change we will provide the exact information about the failure. Such as:
```Node has insufficient capacity```
Will be:
```Node has insufficient capacity for dimension: testKey. Left available: 1, Required: 5```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on PR #2639:
URL: https://github.com/apache/helix/pull/2639#issuecomment-1771839789
This PR is approved by @junkaixue and is ready to be merged.
Final commit message:
In this PR, we are logging the specific details of the hard constraint validation failures, that improvs the issue debuggability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on PR #2639:
URL: https://github.com/apache/helix/pull/2639#issuecomment-1771435497
pls follow the final commit message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] [apache/helix] -- Fixes #2638, Improve Hard Constraint Failure Debuggability by adding details in the error message [helix]
Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2639:
URL: https://github.com/apache/helix/pull/2639#discussion_r1344527862
##########
helix-core/src/main/java/org/apache/helix/controller/rebalancer/waged/constraints/ConstraintBasedAlgorithm.java:
##########
@@ -120,24 +120,24 @@ public OptimalAssignment calculate(ClusterModel clusterModel) throws HelixRebala
private Optional<AssignableNode> getNodeWithHighestPoints(AssignableReplica replica,
List<AssignableNode> assignableNodes, ClusterContext clusterContext,
Set<String> busyInstances, OptimalAssignment optimalAssignment) {
- Map<AssignableNode, List<HardConstraint>> hardConstraintFailures = new ConcurrentHashMap<>();
+ Map<AssignableNode, List<String>> hardConstraintFailures = new ConcurrentHashMap<>();
List<AssignableNode> candidateNodes = assignableNodes.parallelStream().filter(candidateNode -> {
boolean isValid = true;
// need to record all the failure reasons and it gives us the ability to debug/fix the runtime
// cluster environment
for (HardConstraint hardConstraint : _hardConstraints) {
- if (!hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext)) {
+ ValidationResult validationResult = hardConstraint.isAssignmentValid(candidateNode, replica, clusterContext);
Review Comment:
changing method signature is not recommended (as this is used as library and ppl may use that API)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org