You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "GrantPSpencer (via GitHub)" <gi...@apache.org> on 2025/01/13 19:35:25 UTC
[PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
GrantPSpencer opened a new pull request, #2994:
URL: https://github.com/apache/helix/pull/2994
### Issues
- [ ] My PR addresses the following Helix issues and references them in the PR description:
Feature warranted by discussion from PR #2932
Discussion here: https://github.com/apache/helix/pull/2932#discussion_r1868312803
### Description
- [ ] Here are some details about my PR, including screenshots of any UI changes:
This PR adds an ATOMIC recursive delete method and uses it for dropping/purging instances from the cluster. The deleteRecursivelyAtomic method will either fully fail or fully succeed. This will prevent partially deleted instances from entering the cluster into an invalid state. This behaves similarly to the current deleteRecursively method, but with the addition of atomicity. There are two signatures available, one for deleting a single path and its children, and one for deleting multiple paths and all their children. Atomicity across different paths is needed to ensure atomicity of dropping an instance.
The FederatedZkClient implementation throws an IllegalArgumentException when deleteRecursivelyAtomic is called upon 2 paths that exist in different ZK realms as atomicity cannot be guaranteed across different realms due to their being 2 separate ZKClients and therefore 2 separate calls.
### Tests
- [ ] The following tests are written for this issue:
testDeleteRecursivelyAtomic in TestRawZkClient.java
- The following is the result of the "mvn test" command on the appropriate module:
Ran manual CI on my personal fork and saw no new failing tests introduced.
### Changes that Break Backward Compatibility (Optional)
- My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
N/A
### Commits
- My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1919620772
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1826,61 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ deleteRecursivelyAtomic(Arrays.asList(path));
+ }
+
+ /**
+ * Delete the paths as well as all their children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths.
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
Review Comment:
1. I feel like we should check the multi result. Even if there are exception thrown, multi may or may not succeeded.
2. I think having specific exception class might be better comparing to do a global catch here, and log corresponding error messages . For example, if we get a `InterruptedException`, we know there is concurrent edit of the paths.
related doc:
OpResult: https://www.javadoc.io/static/org.apache.zookeeper/zookeeper/3.7.1/org/apache/zookeeper/OpResult.DeleteResult.html
Multi(): https://www.javadoc.io/doc/org.apache.zookeeper/zookeeper/3.7.1/org/apache/zookeeper/ZooKeeper.html#multi-java.lang.Iterable-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2994:
URL: https://github.com/apache/helix/pull/2994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1913982807
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1825,74 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
+ }
+
+ /**
+ * Delete all provided paths as well as all their children. This operation is atomic and will either delete all nodes
+ * or none. This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete paths {}, exception {}", _uid, paths, e);
+ throw new ZkClientException("Failed to delete paths " + paths, e);
+ }
+ }
+
+ /**
+ * Get the list of operations to delete the given root and all its children. Performs simple BFS to put delete
+ * operations for leaf nodes first before parent nodes.
+ * @param root the root node to delete
+ * @return the list of ZK operations to delete the given root and all its children
+ */
+ private List<Op> getOpsForRecursiveDelete(String root) {
+ List<Op> ops = new ArrayList<>();
+ // Return early if the root does not exist
+ if (!exists(root)) {
+ return ops;
+ }
+
+ HashSet<String> visited = new HashSet<>();
Review Comment:
Changed algo. Goal is to add end nodes first so that all in the ordered list of ops, the op for deleting a parent node comes after the ops for deleting all of its children
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1913982070
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1825,74 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
+ }
+
+ /**
+ * Delete all provided paths as well as all their children. This operation is atomic and will either delete all nodes
+ * or none. This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete paths {}, exception {}", _uid, paths, e);
+ throw new ZkClientException("Failed to delete paths " + paths, e);
+ }
+ }
+
+ /**
+ * Get the list of operations to delete the given root and all its children. Performs simple BFS to put delete
+ * operations for leaf nodes first before parent nodes.
+ * @param root the root node to delete
+ * @return the list of ZK operations to delete the given root and all its children
+ */
+ private List<Op> getOpsForRecursiveDelete(String root) {
+ List<Op> ops = new ArrayList<>();
+ // Return early if the root does not exist
+ if (!exists(root)) {
+ return ops;
+ }
+
+ HashSet<String> visited = new HashSet<>();
+ Stack<String> nodes = new Stack<>();
+ nodes.push(root);
+
+ while (!nodes.isEmpty()) {
+ String node = nodes.peek();
+ List<String> children = getChildren(node, false);
+ if (children.isEmpty() || visited.contains(node)) {
+ nodes.pop();
+ ops.add(Op.delete(node, -1));
Review Comment:
Children need to come first in the ordered list of ops. From my testing, ZK server applies these sequentially so deletion of parent coming before deletion of all children will throw directory not empty exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1913855031
##########
helix-core/src/test/java/org/apache/helix/manager/zk/TestZkBaseDataAccessor.java:
##########
@@ -49,6 +49,7 @@
import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.Test;
+import org.apache.zookeeper.Op;
Review Comment:
No code change why need to add this?
##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java:
##########
@@ -274,18 +273,14 @@ public void dropInstance(String clusterName, InstanceConfig instanceConfig) {
"Node " + instanceName + " is still alive for cluster " + clusterName + ", can't drop.");
}
- // delete config path
- String instanceConfigsPath = PropertyPathBuilder.instanceConfig(clusterName);
- ZKUtil.dropChildren(_zkClient, instanceConfigsPath, instanceConfig.getRecord());
- // delete instance path
- dropInstancePathRecursively(instancePath, instanceConfig.getInstanceName());
+ dropInstancePathsRecursively(instanceName, instancePath, instanceConfigPath);
Review Comment:
I dont think we allow instance config name under config different from what stays in INSTANCE folder. This is required in Helix.
So delete it by default assumption without give flexibility to add path unless it is general delete with extra path.
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1825,74 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
+ }
+
+ /**
+ * Delete all provided paths as well as all their children. This operation is atomic and will either delete all nodes
+ * or none. This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete paths {}, exception {}", _uid, paths, e);
+ throw new ZkClientException("Failed to delete paths " + paths, e);
+ }
+ }
+
+ /**
+ * Get the list of operations to delete the given root and all its children. Performs simple BFS to put delete
+ * operations for leaf nodes first before parent nodes.
+ * @param root the root node to delete
+ * @return the list of ZK operations to delete the given root and all its children
+ */
+ private List<Op> getOpsForRecursiveDelete(String root) {
+ List<Op> ops = new ArrayList<>();
+ // Return early if the root does not exist
+ if (!exists(root)) {
+ return ops;
+ }
+
+ HashSet<String> visited = new HashSet<>();
+ Stack<String> nodes = new Stack<>();
+ nodes.push(root);
+
+ while (!nodes.isEmpty()) {
+ String node = nodes.peek();
+ List<String> children = getChildren(node, false);
+ if (children.isEmpty() || visited.contains(node)) {
+ nodes.pop();
+ ops.add(Op.delete(node, -1));
Review Comment:
Also a question here. If the parent node add first, will it cause problem with deletion failed...
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1825,74 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
+ }
+
+ /**
+ * Delete all provided paths as well as all their children. This operation is atomic and will either delete all nodes
+ * or none. This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete paths {}, exception {}", _uid, paths, e);
+ throw new ZkClientException("Failed to delete paths " + paths, e);
+ }
+ }
+
+ /**
+ * Get the list of operations to delete the given root and all its children. Performs simple BFS to put delete
+ * operations for leaf nodes first before parent nodes.
+ * @param root the root node to delete
+ * @return the list of ZK operations to delete the given root and all its children
+ */
+ private List<Op> getOpsForRecursiveDelete(String root) {
+ List<Op> ops = new ArrayList<>();
+ // Return early if the root does not exist
+ if (!exists(root)) {
+ return ops;
+ }
+
+ HashSet<String> visited = new HashSet<>();
Review Comment:
If it is a tree not a cycled graph and BFS does not require visited to check....
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1825,74 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
+ }
+
+ /**
+ * Delete all provided paths as well as all their children. This operation is atomic and will either delete all nodes
+ * or none. This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete paths {}, exception {}", _uid, paths, e);
+ throw new ZkClientException("Failed to delete paths " + paths, e);
+ }
+ }
+
+ /**
+ * Get the list of operations to delete the given root and all its children. Performs simple BFS to put delete
+ * operations for leaf nodes first before parent nodes.
+ * @param root the root node to delete
+ * @return the list of ZK operations to delete the given root and all its children
+ */
+ private List<Op> getOpsForRecursiveDelete(String root) {
+ List<Op> ops = new ArrayList<>();
+ // Return early if the root does not exist
+ if (!exists(root)) {
+ return ops;
+ }
+
+ HashSet<String> visited = new HashSet<>();
+ Stack<String> nodes = new Stack<>();
+ nodes.push(root);
+
+ while (!nodes.isEmpty()) {
+ String node = nodes.peek();
+ List<String> children = getChildren(node, false);
+ if (children.isEmpty() || visited.contains(node)) {
+ nodes.pop();
+ ops.add(Op.delete(node, -1));
+ } else {
+ for (String child : children) {
+ nodes.push(node + "/" + child);
+ }
+ }
+ visited.add(node);
Review Comment:
Too messy...
String node = nodes.pop();
List<String> children = getChildren(node, false);
if (children.isEmpty()) {
ops.add(Op.delete(node, -1));
continue;
}
children.stream().foreach(c -> nodes.push(node + "/" + c);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1917327288
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1826,66 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
+ }
+
+ /**
+ * Delete all provided paths as well as all their children. This operation is atomic and will either delete all nodes
+ * or none. This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete paths {}, exception {}", _uid, paths, e);
+ throw new ZkClientException("Failed to delete paths " + paths, e);
+ }
+ }
+
+ /**
+ * Get the list of operations to delete the given root and all its children. Ops will be ordered so that deletion of
+ * children will come before parent nodes.
+ * @param root the root node to delete
+ * @return the list of ZK operations to delete the given root and all its children
+ */
+ private List<Op> getOpsForRecursiveDelete(String root) {
+ List<Op> ops = new ArrayList<>();
+ // Return early if the root does not exist
+ if (!exists(root)) {
+ return ops;
+ }
+
+ Queue<String> nodes = new LinkedList<>();
+ nodes.offer(root);
+ while (!nodes.isEmpty()) {
+ String node = nodes.poll();
+ getChildren(node, false).stream().forEach(child -> nodes.offer(node + "/" + child));
+ ops.add(Op.delete(node, -1));
+ }
+
+ Collections.reverse(ops);
Review Comment:
Add a comment here that this sorting is necessary.
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1826,66 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
Review Comment:
I prefer not have branch of logics.
This code can leverage deleteRecursivelyAtomic(List<String> paths) for implementation.
like
deleteRecursivelyAtomic(Arrays.asList(path));
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on PR #2994:
URL: https://github.com/apache/helix/pull/2994#issuecomment-2611179547
Generally LGTM. Please address final comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on PR #2994:
URL: https://github.com/apache/helix/pull/2994#issuecomment-2616957270
Pull request approved by: @xyuanlu
Commit message: Add atomic recursive delete to ZK client and use for drop instance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1917511134
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1826,66 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
+ }
+
+ /**
+ * Delete all provided paths as well as all their children. This operation is atomic and will either delete all nodes
+ * or none. This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete paths {}, exception {}", _uid, paths, e);
+ throw new ZkClientException("Failed to delete paths " + paths, e);
+ }
+ }
+
+ /**
+ * Get the list of operations to delete the given root and all its children. Ops will be ordered so that deletion of
+ * children will come before parent nodes.
+ * @param root the root node to delete
+ * @return the list of ZK operations to delete the given root and all its children
+ */
+ private List<Op> getOpsForRecursiveDelete(String root) {
+ List<Op> ops = new ArrayList<>();
+ // Return early if the root does not exist
+ if (!exists(root)) {
+ return ops;
+ }
+
+ Queue<String> nodes = new LinkedList<>();
+ nodes.offer(root);
+ while (!nodes.isEmpty()) {
+ String node = nodes.poll();
+ getChildren(node, false).stream().forEach(child -> nodes.offer(node + "/" + child));
+ ops.add(Op.delete(node, -1));
+ }
+
+ Collections.reverse(ops);
Review Comment:
Updated. Added comment for tree traversal and list reversal
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1826,66 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ List<Op> ops = getOpsForRecursiveDelete(path);
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
+ LOG.error("zkclient {}, Failed to delete {}, exception {}", _uid, path, e);
+ throw new ZkClientException("Failed to delete " + path, e);
+ }
Review Comment:
Thank you for the feedback. Udpated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1913980769
##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java:
##########
@@ -274,18 +273,14 @@ public void dropInstance(String clusterName, InstanceConfig instanceConfig) {
"Node " + instanceName + " is still alive for cluster " + clusterName + ", can't drop.");
}
- // delete config path
- String instanceConfigsPath = PropertyPathBuilder.instanceConfig(clusterName);
- ZKUtil.dropChildren(_zkClient, instanceConfigsPath, instanceConfig.getRecord());
- // delete instance path
- dropInstancePathRecursively(instancePath, instanceConfig.getInstanceName());
+ dropInstancePathsRecursively(instanceName, instancePath, instanceConfigPath);
Review Comment:
Good call out. Changed to take clusterName and instanceName and now constructs paths accordingly. There is some redundancy in the string construction of paths now, but will prevent possibility of passing instancePath of 1 instance and instanceConfigPath of a different instance.
only zkclient should have general delete with any paths
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1924485199
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1826,61 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ deleteRecursivelyAtomic(Arrays.asList(path));
+ }
+
+ /**
+ * Delete the paths as well as all their children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths.
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
+ multi(ops);
+ }
+ catch (Exception e) {
Review Comment:
1. Added check for any ErrorResults in response from multi() call. The class only provides int values that correspond to specific ZK KeeperExceptions. We can convert that int value to a KeeperException.Code and provide a list of when we log + throw error.
2. The goal with the global catch was to follow the same pattern as the original deleteRecursively method which global catches and throws a ZkClientException. Do you think specific error catching is necessary in this case? My understanding is that the original exception type (lets say InterruptedException) will still be printed to log and the cause will still be preserved in the `cause` of the new ZkClientException. The main draw back is that callers of this API will not be able to handle specific failures themselves, but will only be able to catch and respond to ZkClientException
Thank you for the feedback @xyuanlu 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [PR] Add atomic recursive delete to ZK client and use for drop instance [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2994:
URL: https://github.com/apache/helix/pull/2994#discussion_r1927786226
##########
zookeeper-api/src/main/java/org/apache/helix/zookeeper/zkclient/ZkClient.java:
##########
@@ -1820,6 +1826,75 @@ public void deleteRecursively(String path) throws ZkClientException {
}
}
+ /**
+ * Delete the path as well as all its children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under the path.
+ * @param path ZK path to delete
+ */
+ public void deleteRecursivelyAtomic(String path) {
+ deleteRecursivelyAtomic(Arrays.asList(path));
+ }
+
+ /**
+ * Delete the paths as well as all their children. This operation is atomic and will either delete all nodes or none.
+ * This operation may fail if another agent is concurrently creating or deleting nodes under any of the paths.
+ * @param paths ZK paths to delete
+ */
+ public void deleteRecursivelyAtomic(List<String> paths) {
+ List<Op> ops = new ArrayList<>();
+ List<OpResult> opResults;
+ for (String path : paths) {
+ ops.addAll(getOpsForRecursiveDelete(path));
+ }
+ try {
Review Comment:
Add special handling for
catch (ZkNoNodeException e) {
// if the node to be deleted does not exist, treat it as success.
return;
}
So that we have feature parity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org