You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/11/05 09:17:48 UTC
[GitHub] [ozone] JacksonYao287 opened a new pull request #2808: HDDS-5517. Support multiple container moves from a source datanode
JacksonYao287 opened a new pull request #2808:
URL: https://github.com/apache/ozone/pull/2808
## What changes were proposed in this pull request?
Support multiple container moves from a source datanode
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5517
## How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752884442
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceGreedy.java
##########
@@ -49,7 +49,11 @@
double currentUsageOfB = b.calculateUtilization(
-sizeLeavingNode.get(b.getDatanodeDetails()));
//in descending order
- return Double.compare(currentUsageOfB, currentUsageOfA);
+ int ret = Double.compare(currentUsageOfB, currentUsageOfA);
+ if (ret != 0) {
+ return ret;
+ }
+ return a.hashCode() - b.hashCode();
Review comment:
sure, will do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-973888931
@JacksonYao287 We will have to get a clean CI before merge. That is the process we follow for Ozone. Maybe you can try rebasing the PR on current master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r773675544
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceGreedy.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.balancer;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.node.DatanodeUsageInfo;
+import org.apache.hadoop.hdds.scm.node.NodeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.UUID;
+
+/**
+ * The selection criteria for selecting source datanodes , the containers of
+ * which will be moved out.
+ */
+public class FindSourceGreedy implements FindSourceStrategy{
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FindSourceGreedy.class);
+ private Map<DatanodeDetails, Long> sizeLeavingNode;
+ private PriorityQueue<DatanodeUsageInfo> potentialSources;
+ private NodeManager nodeManager;
+ private ContainerBalancerConfiguration config;
+ private Double lowerLimit;
+
+ FindSourceGreedy(NodeManager nodeManager) {
+ sizeLeavingNode = new HashMap<>();
+ potentialSources = new PriorityQueue<>((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ -sizeLeavingNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ -sizeLeavingNode.get(b.getDatanodeDetails()));
+ //in descending order
+ int ret = Double.compare(currentUsageOfB, currentUsageOfA);
+ if (ret != 0) {
+ return ret;
+ }
+ UUID uuidA = a.getDatanodeDetails().getUuid();
+ UUID uuidB = b.getDatanodeDetails().getUuid();
+ return uuidA.compareTo(uuidB);
+ });
+ this.nodeManager = nodeManager;
+ }
+
+ private void setLowerLimit(Double lowerLimit) {
+ this.lowerLimit = lowerLimit;
+ }
+
+ private void setPotentialSources(
+ List<DatanodeUsageInfo> potentialSourceDataNodes) {
+ potentialSources.clear();
+ sizeLeavingNode.clear();
+ potentialSourceDataNodes.forEach(
+ c -> sizeLeavingNode.put(c.getDatanodeDetails(), 0L));
+ potentialSources.addAll(potentialSourceDataNodes);
+ }
+
+ private void setConfiguration(ContainerBalancerConfiguration conf) {
+ this.config = conf;
+ }
+
+ /**
+ * increase the Leaving size of a candidate source data node.
+ */
+ @Override
+ public void increaseSizeLeaving(DatanodeDetails dui, long size) {
+ Long currentSize = sizeLeavingNode.get(dui);
+ if(currentSize != null) {
+ sizeLeavingNode.put(dui, currentSize + size);
+ //reorder according to the latest sizeLeavingNode
+ potentialSources.add(nodeManager.getUsageInfo(dui));
+ return;
Review comment:
Hey @JacksonYao287 Can you help me understand what's happening in this method? I don't think the usage info for a node will get updated during an iteration, since DU/DF don't run during an iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r773826918
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceGreedy.java
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.balancer;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.node.DatanodeUsageInfo;
+import org.apache.hadoop.hdds.scm.node.NodeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.UUID;
+
+/**
+ * The selection criteria for selecting source datanodes , the containers of
+ * which will be moved out.
+ */
+public class FindSourceGreedy implements FindSourceStrategy{
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FindSourceGreedy.class);
+ private Map<DatanodeDetails, Long> sizeLeavingNode;
+ private PriorityQueue<DatanodeUsageInfo> potentialSources;
+ private NodeManager nodeManager;
+ private ContainerBalancerConfiguration config;
+ private Double lowerLimit;
+
+ FindSourceGreedy(NodeManager nodeManager) {
+ sizeLeavingNode = new HashMap<>();
+ potentialSources = new PriorityQueue<>((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ -sizeLeavingNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ -sizeLeavingNode.get(b.getDatanodeDetails()));
+ //in descending order
+ int ret = Double.compare(currentUsageOfB, currentUsageOfA);
+ if (ret != 0) {
+ return ret;
+ }
+ UUID uuidA = a.getDatanodeDetails().getUuid();
+ UUID uuidB = b.getDatanodeDetails().getUuid();
+ return uuidA.compareTo(uuidB);
+ });
+ this.nodeManager = nodeManager;
+ }
+
+ private void setLowerLimit(Double lowerLimit) {
+ this.lowerLimit = lowerLimit;
+ }
+
+ private void setPotentialSources(
+ List<DatanodeUsageInfo> potentialSourceDataNodes) {
+ potentialSources.clear();
+ sizeLeavingNode.clear();
+ potentialSourceDataNodes.forEach(
+ c -> sizeLeavingNode.put(c.getDatanodeDetails(), 0L));
+ potentialSources.addAll(potentialSourceDataNodes);
+ }
+
+ private void setConfiguration(ContainerBalancerConfiguration conf) {
+ this.config = conf;
+ }
+
+ /**
+ * increase the Leaving size of a candidate source data node.
+ */
+ @Override
+ public void increaseSizeLeaving(DatanodeDetails dui, long size) {
+ Long currentSize = sizeLeavingNode.get(dui);
+ if(currentSize != null) {
+ sizeLeavingNode.put(dui, currentSize + size);
+ //reorder according to the latest sizeLeavingNode
+ potentialSources.add(nodeManager.getUsageInfo(dui));
+ return;
Review comment:
let me explain this.
when selecting a source datanode, we always want to the select the one which has a largest storage usage. here, i use `PriorityQueue`, which is fast to get the top one. when calling `getNextCandidateSourceDataNode`, `PriorityQueue#poll` is called , which will get and remove the top source data node for this PriorityQueue.
actually, now we supporting move multiple containers from one datanode , so a data node can be selected as source for multiple times in one iteration.
here are two reasons to call `potentialSources.add(nodeManager.getUsageInfo(dui));`
1 add the data node back to the PriorityQueue again, so it can be selected as a source again.
2 when we update `sizeLeavingNode`, the usage of this data node will be considered changed (the reported usage - sizeLeaving), so we need to sort all the candidate source datanodes according to the latest usage and get the top one. when adding the data node back to the PriorityQueue, PriorityQueue will sort all the datanode again(it will use heap sort , so very fast), so we can get the next top one.
i am not sure is it clear to you now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-969835559
@lokeshj1703 @siddhantsangwan i have refactored the patch according to the comments, please take a look. if this looks good to you , i will improve the Code comments in the a commit.
>in some case , one container may be moved from a withinThresholdUtilized node to another withinThresholdUtilized node
i will create a separate patch to add withinThreshold nodes back into candidate target and source datanodes, and fix the potential bug in the newly refactored code. i think after refactoring, we can do this more gracefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-968784537
thanks @lokeshj1703 and @siddhantsangwan for the review!
>1. I was thinking we should have sth similar to FindTargetStrategy for source like FindSourceStrategy.
we can add an interface of FindSourceStrategy and add a default implementation for it. this can be done by refactoring `SourceDataNodeSelectionCriteria`.
>2. Let's have a separate PR for withinThreshold nodes removal. I think there was a use case where accounting them was important where there are 5 over utilized and 5 within threshold nodes.
sure, let us create another to solve this, and fix related bug. in this patch , we focus supporting multiple container moves from a source datanode. i will add withinThreshold nodes back in a new commit
>3. Also by default we shouldn't allow all size of data to be moved from source/target. It is important to limit otherwise it is possible for all balancing to happen in 2-4 nodes.
in this patch , before matching a target with a source , we will sort all the target datanodes In ascending order and all the source datanodes in descending order by usage rate considering `SizeEnteringnode` and `SizeLeavingNode`, so that we always try to match a source datanode with the biggest usage to the target datanode with a smallest usage. it is Almost impossible for all balancing to happen in only 2-4 nodes， unless those 2-4 nodes has a much bigger or smaller usage than others. i think if that case happens, it makes sense to move only among these Very unbalanced datanodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r751160266
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceGreedy.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.balancer;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.node.DatanodeUsageInfo;
+import org.apache.hadoop.hdds.scm.node.NodeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The selection criteria for selecting source datanodes , the containers of
+ * which will be moved out.
+ */
+public class FindSourceGreedy implements FindSourceStrategy{
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FindSourceGreedy.class);
+ private Map<DatanodeDetails, Long> sizeLeavingNode;
+ private List<DatanodeUsageInfo> potentialSources;
+ private NodeManager nodeManager;
+ private ContainerBalancerConfiguration config;
+ private Double lowerLimit;
+
+ FindSourceGreedy(NodeManager nodeManager) {
+ sizeLeavingNode = new HashMap<>();
+ this.nodeManager = nodeManager;
+ }
+
+ private void setLowerLimit(Double lowerLimit) {
+ this.lowerLimit = lowerLimit;
+ }
+
+ private void setPotentialSources(
+ List<DatanodeUsageInfo> potentialSources) {
+ this.potentialSources = potentialSources;
+ sizeLeavingNode.clear();
+ potentialSources.forEach(
+ c -> sizeLeavingNode.put(c.getDatanodeDetails(), 0L));
+ }
+
+ private void setConfiguration(ContainerBalancerConfiguration conf) {
+ this.config = conf;
+ }
+
+ /**
+ * increase the Leaving size of a candidate source data node.
+ */
+ @Override
+ public void increaseSizeLeaving(DatanodeDetails dui, long size) {
+ Long currentSize = sizeLeavingNode.get(dui);
+ if(currentSize != null) {
+ sizeLeavingNode.put(dui, currentSize + size);
+ return;
+ }
+ LOG.warn("Cannot find datanode {} in candidate source datanodes",
+ dui.getUuid());
+ }
+
+ /**
+ * get the next candidate source data node according to
+ * the strategy.
+ *
+ * @return the nex candidate source data node.
+ */
+ @Override
+ public DatanodeDetails getNextCandidateSourceDataNode() {
+ if (potentialSources.isEmpty()) {
+ LOG.info("no more candidate source data node");
+ return null;
+ }
+ //TODO：use a more quick data structure, which will hava a
+ // better performance when changing or deleting one element at once
+ potentialSources.sort((a, b) -> {
Review comment:
TreeSet is implemented by a binary search tree. when searching or inserting any element , it will take O(logN) of time.
priority queue is implemented with heap sort, which is very quick to get the top 1 element after initialized, and I have considered using it。
here, we always try to get the element with the biggest usage(top 1). so i think it is better off using priority queue here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 merged pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 merged pull request #2808:
URL: https://github.com/apache/ozone/pull/2808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-976222534
thanks @lokeshj1703 and @siddhantsangwan for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-966943333
thanks very much @siddhantsangwan for the review, i have updated the patch , please take a look!
>By removing withinThresholdUtilizedNodes, we rely on the user setting a suitable threshold to make balancing work in such cases.
i do get your point @Siddhant Sangwan. i think the main difference between our ideas is what should balancer exactly do.
in your opinion, balancer should always try its best to make the cluster more balanced，no matter what the threshold is.
but in mine, balancer just try to make the cluster balanced to what the threshold specified, if we want the cluster to be more balanced, we should specified a smaller threshold.
on one hand, in practice, i think it easy to specified a smaller one if we find current threshold does not take any effect.
on the other hand, if we want the cluster to be more balanced , we can not rely on balancer`s effort with a big threshold, because how balanced balancer will make the cluster to be is uncertain. specifying a smaller threshold will definitely work as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-969835559
@lokeshj1703 @siddhantsangwan i have refactored the patch according to the comments, please take a look. if this looks good to you , i will improve the Code comments in a new commit.
>in some case , one container may be moved from a withinThresholdUtilized node to another withinThresholdUtilized node
i will create a separate patch to add withinThreshold nodes back into candidate target and source datanodes, and fix the potential bug in the newly refactored code. i think after refactoring, we can do this more gracefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-972786786
@lokeshj1703 thanks for pointing out the mistake , i have updated this patch , please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r751965087
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -44,37 +46,64 @@
private ContainerManager containerManager;
private PlacementPolicy placementPolicy;
+ private Map<DatanodeDetails, Long> sizeEnteringNode;
+ private NodeManager nodeManager;
+ private ContainerBalancerConfiguration config;
+ private Double upperLimit;
+ private List<DatanodeUsageInfo> potentialTargets;
public FindTargetGreedy(
ContainerManager containerManager,
- PlacementPolicy placementPolicy) {
+ PlacementPolicy placementPolicy,
+ NodeManager nodeManager) {
+ sizeEnteringNode = new HashMap<>();
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
+ this.nodeManager = nodeManager;
+ }
+
+ private void setUpperLimit(Double upperLimit){
+ this.upperLimit = upperLimit;
+ }
+
+ private void setPotentialTargets(List<DatanodeUsageInfo> potentialTargets) {
+ this.potentialTargets = potentialTargets;
+ sizeEnteringNode.clear();
+ potentialTargets.forEach(
+ p -> sizeEnteringNode.put(p.getDatanodeDetails(), 0L));
+ }
+
+ private void setConfiguration(ContainerBalancerConfiguration conf) {
+ this.config = conf;
}
/**
* Find a {@link ContainerMoveSelection} consisting of a target and
* container to move for a source datanode. Favours more under-utilized nodes.
* @param source Datanode to find a target for
- * @param potentialTargets Collection of potential target datanodes
* @param candidateContainers Set of candidate containers satisfying
* selection criteria
* {@link ContainerBalancerSelectionCriteria}
- * @param canSizeEnterTarget A functional interface whose apply
* (DatanodeDetails, Long) method returns true if the size specified in the
* second argument can enter the specified DatanodeDetails node
* @return Found target and container
*/
@Override
public ContainerMoveSelection findTargetForContainerMove(
- DatanodeDetails source, Collection<DatanodeDetails> potentialTargets,
- Set<ContainerID> candidateContainers,
- BiFunction<DatanodeDetails, Long, Boolean> canSizeEnterTarget) {
- for (DatanodeDetails target : potentialTargets) {
+ DatanodeDetails source, Set<ContainerID> candidateContainers) {
+ potentialTargets.sort((a, b) -> {
Review comment:
Let's do a priority queue implementation here as well.
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerConfiguration.java
##########
@@ -75,17 +73,15 @@
defaultValue = "", tags = {ConfigTag.BALANCER}, description = "The " +
"maximum size that can enter a target datanode in each " +
"iteration while balancing. This is the sum of data from multiple " +
- "sources. The default value is greater than the configured" +
- " (or default) ozone.scm.container.size by 1GB.")
- private long maxSizeEnteringTarget;
+ "sources. by default, we do not limit this value")
+ private long maxSizeEnteringTarget = Long.MAX_VALUE;
Review comment:
Let's revert the configuration changes here and do it in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-973884484
@lokeshj1703 @siddhantsangwan thank you for the review! CI failure seems not caused by this patch.
i have tested this patch in my Kubernetes cluster, and it works as expected。the process of balancing is greatly accelerated。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-969835559
@lokeshj1703 @siddhantsangwan i have refactored the patch according to the comments, please take a look. if this looks good to you , i will improve the Code comments in a new commit.
>in some case , one container may be moved from a withinThresholdUtilized node to another withinThresholdUtilized node
i will create a separate patch to add withinThreshold nodes back into candidate target and source datanodes, and fix the potential bug of withinThreshold nodes. i think after refactoring, we can do this more gracefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-976215025
@JacksonYao287 Thanks for the contribution! @siddhantsangwan Thanks for review! I have committed the PR to master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r744460061
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -93,6 +94,7 @@
private Map<DatanodeDetails, ContainerMoveSelection> sourceToTargetMap;
private Map<DatanodeDetails, Long> sizeLeavingNode;
private Map<DatanodeDetails, Long> sizeEnteringNode;
+ private Map<DatanodeDetails, Long> remainingUsageBeforeOverUtilized;
Review comment:
I think we should not add this map `remainingUsageBeforeOverUtilized`. We can probably use sizeLeavingNode here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r751104203
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceGreedy.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.balancer;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.node.DatanodeUsageInfo;
+import org.apache.hadoop.hdds.scm.node.NodeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The selection criteria for selecting source datanodes , the containers of
+ * which will be moved out.
+ */
+public class FindSourceGreedy implements FindSourceStrategy{
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FindSourceGreedy.class);
+ private Map<DatanodeDetails, Long> sizeLeavingNode;
+ private List<DatanodeUsageInfo> potentialSources;
+ private NodeManager nodeManager;
+ private ContainerBalancerConfiguration config;
+ private Double lowerLimit;
+
+ FindSourceGreedy(NodeManager nodeManager) {
+ sizeLeavingNode = new HashMap<>();
+ this.nodeManager = nodeManager;
+ }
+
+ private void setLowerLimit(Double lowerLimit) {
+ this.lowerLimit = lowerLimit;
+ }
+
+ private void setPotentialSources(
+ List<DatanodeUsageInfo> potentialSources) {
+ this.potentialSources = potentialSources;
+ sizeLeavingNode.clear();
+ potentialSources.forEach(
+ c -> sizeLeavingNode.put(c.getDatanodeDetails(), 0L));
+ }
+
+ private void setConfiguration(ContainerBalancerConfiguration conf) {
+ this.config = conf;
+ }
+
+ /**
+ * increase the Leaving size of a candidate source data node.
+ */
+ @Override
+ public void increaseSizeLeaving(DatanodeDetails dui, long size) {
+ Long currentSize = sizeLeavingNode.get(dui);
+ if(currentSize != null) {
+ sizeLeavingNode.put(dui, currentSize + size);
+ return;
+ }
+ LOG.warn("Cannot find datanode {} in candidate source datanodes",
+ dui.getUuid());
+ }
+
+ /**
+ * get the next candidate source data node according to
+ * the strategy.
+ *
+ * @return the nex candidate source data node.
+ */
+ @Override
+ public DatanodeDetails getNextCandidateSourceDataNode() {
+ if (potentialSources.isEmpty()) {
+ LOG.info("no more candidate source data node");
+ return null;
+ }
+ //TODO：use a more quick data structure, which will hava a
+ // better performance when changing or deleting one element at once
+ potentialSources.sort((a, b) -> {
Review comment:
We can probably use a priority queue here or a TreeSet. TreeSet might be better.
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -518,14 +475,31 @@ private ContainerMoveSelection matchSourceWithTarget(
}
return null;
}
+
+ //if the utilization of the source data node becomes lower than lowerLimit
+ //after the container is moved out , then the container can not be
+ // a candidate one, and we should remove it from the candidateContainers.
+ candidateContainers.removeIf(c -> {
+ ContainerInfo cInfo;
+ try {
+ cInfo = containerManager.getContainer(c);
+ } catch (ContainerNotFoundException e) {
+ LOG.warn("Could not find container {} when " +
+ "be matched with a move target", c);
+ //remove this not found container
+ return true;
+ }
+ return !findSourceStrategy.canSizeLeaveSource(
+ source, cInfo.getUsedBytes());
+ });
Review comment:
Can we move this to selection criteria#getCandidateContainers? Maybe selection criteria can have reference to find source strategy.
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -341,99 +341,59 @@ private boolean initializeIteration() {
nodeManager, replicationManager, containerManager);
sourceToTargetMap = new HashMap<>(overUtilizedNodes.size() +
withinThresholdUtilizedNodes.size());
-
- // initialize maps to track how much size is leaving and entering datanodes
- sizeLeavingNode = new HashMap<>(overUtilizedNodes.size() +
- withinThresholdUtilizedNodes.size());
- overUtilizedNodes.forEach(datanodeUsageInfo -> sizeLeavingNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
- withinThresholdUtilizedNodes.forEach(datanodeUsageInfo -> sizeLeavingNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
-
- sizeEnteringNode = new HashMap<>(underUtilizedNodes.size() +
- withinThresholdUtilizedNodes.size());
- underUtilizedNodes.forEach(datanodeUsageInfo -> sizeEnteringNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
- withinThresholdUtilizedNodes.forEach(datanodeUsageInfo -> sizeEnteringNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
-
return true;
}
private IterationResult doIteration() {
// note that potential and selected targets are updated in the following
// loop
- List<DatanodeDetails> potentialTargets = getPotentialTargets();
+ //TODO(jacksonyao): take withinThresholdUtilizedNodes as candidate for both
+ // source and target
+ findSourceStrategy.reInitialize(getPotentialSources(), config, lowerLimit);
+ List<DatanodeUsageInfo> potentialTargets = getPotentialTargets();
+ findTargetStrategy.reInitialize(potentialTargets, config, upperLimit);
Review comment:
Can we avoid calling reinitialize in every iteration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-961761852
@lokeshj1703 @siddhantsangwan please take a look at this jira as a priority , i think it will speed up the move process very much.
by the way , i think we should not limit `maxSizeEnteringTarget` and `maxSizeLeavingSource` by default, we can specify the value in the command line if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-966897707
Also, please note the following concern that was raised [here](https://github.com/apache/ozone/pull/2230#discussion_r630705685) for an earlier version of balancer.
> Question:
>
> say we have a 10 DN cluster, the usage of all of them is 95%, then one empty DN is added to rebalance the cluster. Given the threshold is 10%, it seems the balancer will not work in this case, since that 10 DN will not achieve upperLimit.
>
> Have we consider corner case like this ?
By removing `withinThresholdUtilizedNodes`, we rely on the user setting a suitable threshold to make balancing work in such cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-968784537
thanks @lokeshj1703 and @siddhantsangwan for the review!
>1. I was thinking we should have sth similar to FindTargetStrategy for source like FindSourceStrategy.
we can add an interface of FindSourceStrategy and add a default implementation for it. this can be done by refactoring `SourceDataNodeSelectionCriteria`.
>2. Let's have a separate PR for withinThreshold nodes removal. I think there was a use case where accounting them was important where there are 5 over utilized and 5 within threshold nodes.
sure, let us create another PR to solve this, and fix related bug. in this patch , we focus supporting multiple container moves from a source datanode. i will add withinThreshold nodes back in a new commit
>3. Also by default we shouldn't allow all size of data to be moved from source/target. It is important to limit otherwise it is possible for all balancing to happen in 2-4 nodes.
in this patch , before matching a target with a source , we will sort all the target datanodes In ascending order and all the source datanodes in descending order by usage rate considering `SizeEnteringnode` and `SizeLeavingNode`, so that we always try to match a source datanode with the biggest usage to the target datanode with a smallest usage. it is Almost impossible for all balancing to happen in only 2-4 nodes， unless those 2-4 nodes has a much bigger or smaller usage than others. i think if that case happens, it makes sense to move only among these Very unbalanced datanodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r751994931
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerConfiguration.java
##########
@@ -75,17 +73,15 @@
defaultValue = "", tags = {ConfigTag.BALANCER}, description = "The " +
"maximum size that can enter a target datanode in each " +
"iteration while balancing. This is the sum of data from multiple " +
- "sources. The default value is greater than the configured" +
- " (or default) ozone.scm.container.size by 1GB.")
- private long maxSizeEnteringTarget;
+ "sources. by default, we do not limit this value")
+ private long maxSizeEnteringTarget = Long.MAX_VALUE;
Review comment:
sure , will do this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752118100
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -60,17 +61,27 @@ public FindTargetGreedy(
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
this.nodeManager = nodeManager;
+
+ potentialTargets = new TreeSet<>((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ sizeEnteringNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ sizeEnteringNode.get(b.getDatanodeDetails()));
+ return Double.compare(currentUsageOfA, currentUsageOfB);
Review comment:
+1 o.w.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r751147870
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -341,99 +341,59 @@ private boolean initializeIteration() {
nodeManager, replicationManager, containerManager);
sourceToTargetMap = new HashMap<>(overUtilizedNodes.size() +
withinThresholdUtilizedNodes.size());
-
- // initialize maps to track how much size is leaving and entering datanodes
- sizeLeavingNode = new HashMap<>(overUtilizedNodes.size() +
- withinThresholdUtilizedNodes.size());
- overUtilizedNodes.forEach(datanodeUsageInfo -> sizeLeavingNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
- withinThresholdUtilizedNodes.forEach(datanodeUsageInfo -> sizeLeavingNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
-
- sizeEnteringNode = new HashMap<>(underUtilizedNodes.size() +
- withinThresholdUtilizedNodes.size());
- underUtilizedNodes.forEach(datanodeUsageInfo -> sizeEnteringNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
- withinThresholdUtilizedNodes.forEach(datanodeUsageInfo -> sizeEnteringNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
-
return true;
}
private IterationResult doIteration() {
// note that potential and selected targets are updated in the following
// loop
- List<DatanodeDetails> potentialTargets = getPotentialTargets();
+ //TODO(jacksonyao): take withinThresholdUtilizedNodes as candidate for both
+ // source and target
+ findSourceStrategy.reInitialize(getPotentialSources(), config, lowerLimit);
+ List<DatanodeUsageInfo> potentialTargets = getPotentialTargets();
+ findTargetStrategy.reInitialize(potentialTargets, config, upperLimit);
Review comment:
> Is this required for recalculating limits and other metadata in a later PR?
yea, it is. so think we can keep it here in this PR, and i will deal with it in the later coming PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752007790
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -44,37 +46,64 @@
private ContainerManager containerManager;
private PlacementPolicy placementPolicy;
+ private Map<DatanodeDetails, Long> sizeEnteringNode;
+ private NodeManager nodeManager;
+ private ContainerBalancerConfiguration config;
+ private Double upperLimit;
+ private List<DatanodeUsageInfo> potentialTargets;
public FindTargetGreedy(
ContainerManager containerManager,
- PlacementPolicy placementPolicy) {
+ PlacementPolicy placementPolicy,
+ NodeManager nodeManager) {
+ sizeEnteringNode = new HashMap<>();
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
+ this.nodeManager = nodeManager;
+ }
+
+ private void setUpperLimit(Double upperLimit){
+ this.upperLimit = upperLimit;
+ }
+
+ private void setPotentialTargets(List<DatanodeUsageInfo> potentialTargets) {
+ this.potentialTargets = potentialTargets;
+ sizeEnteringNode.clear();
+ potentialTargets.forEach(
+ p -> sizeEnteringNode.put(p.getDatanodeDetails(), 0L));
+ }
+
+ private void setConfiguration(ContainerBalancerConfiguration conf) {
+ this.config = conf;
}
/**
* Find a {@link ContainerMoveSelection} consisting of a target and
* container to move for a source datanode. Favours more under-utilized nodes.
* @param source Datanode to find a target for
- * @param potentialTargets Collection of potential target datanodes
* @param candidateContainers Set of candidate containers satisfying
* selection criteria
* {@link ContainerBalancerSelectionCriteria}
- * @param canSizeEnterTarget A functional interface whose apply
* (DatanodeDetails, Long) method returns true if the size specified in the
* second argument can enter the specified DatanodeDetails node
* @return Found target and container
*/
@Override
public ContainerMoveSelection findTargetForContainerMove(
- DatanodeDetails source, Collection<DatanodeDetails> potentialTargets,
- Set<ContainerID> candidateContainers,
- BiFunction<DatanodeDetails, Long, Boolean> canSizeEnterTarget) {
- for (DatanodeDetails target : potentialTargets) {
+ DatanodeDetails source, Set<ContainerID> candidateContainers) {
+ potentialTargets.sort((a, b) -> {
Review comment:
i think we should use TreeSet here.
in `FindSourceGreedy`, we always try to get the top one source, and if no target and be matched with it , we will remove it.
but here, we never remove any target , and just try to find a target with Middle-order traversal。but we can not do traversal on priorityqueue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752117371
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -60,17 +61,27 @@ public FindTargetGreedy(
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
this.nodeManager = nodeManager;
+
+ potentialTargets = new TreeSet<>((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ sizeEnteringNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ sizeEnteringNode.get(b.getDatanodeDetails()));
+ return Double.compare(currentUsageOfA, currentUsageOfB);
Review comment:
Sorry for not bringing this up earlier! But we will need to handle the case in the comparator where utilisation is same for two nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-969835559
@lokeshj1703 @siddhantsangwan i have refactored the patch according to the comments, please take a look.
>in some case , one container may be moved from a withinThresholdUtilized node to another withinThresholdUtilized node
i will create a separate patch to add withinThreshold nodes back into candidate target and source datanodes, and fix the potential bug in the newly refactored code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752117371
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -60,17 +61,27 @@ public FindTargetGreedy(
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
this.nodeManager = nodeManager;
+
+ potentialTargets = new TreeSet<>((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ sizeEnteringNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ sizeEnteringNode.get(b.getDatanodeDetails()));
+ return Double.compare(currentUsageOfA, currentUsageOfB);
Review comment:
Sorry for not bringing this up earlier! But we will need to handle the case in the comparator where utilisation is same for two nodes. Otherwise two nodes with same utilisation can not exist. It would be better to make similar change for source as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r748021058
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -708,8 +696,8 @@ boolean canSizeEnterTarget(DatanodeDetails target, long size) {
//2 current usage of target datanode plus sizeEnteringAfterMove
// is smaller than or equal to upperLimit
return sizeEnteringAfterMove <= config.getMaxSizeEnteringTarget() &&
- nodeManager.getUsageInfo(target)
- .calculateUtilization(sizeEnteringAfterMove) <= upperLimit;
+ Double.compare(nodeManager.getUsageInfo(target)
+ .calculateUtilization(sizeEnteringAfterMove), upperLimit) < 1;
Review comment:
```suggestion
.calculateUtilization(sizeEnteringAfterMove), upperLimit) <= 0;
```
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -262,20 +261,17 @@ private boolean initializeIteration() {
clusterAvgUtilisation);
}
- // under utilized nodes have utilization(that is, used / capacity) less
- // than lower limit
Review comment:
Let's keep this comment as documentation?
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -518,6 +485,27 @@ private ContainerMoveSelection matchSourceWithTarget(
}
return null;
}
+
+ //if the utilization of the source data node becomes lower than lowerLimit
+ //after the container is moved out , then the container can not be
+ // a candidate one, and we should remove it from the candidateContainers.
+ candidateContainers.removeIf(c -> {
+ ContainerInfo cInfo;
+ try {
+ cInfo = containerManager.getContainer(c);
+ } catch (ContainerNotFoundException e) {
+ LOG.warn("Could not find container {} when " +
+ "be matched with a move target", c);
+ //remove this not found container
+ return true;
+ }
+ Long totalLeavingSize = sizeLeavingNode.get(source) +
+ cInfo.getUsedBytes();
+ return Double.compare(nodeManager.getUsageInfo(source)
+ .calculateUtilization(totalLeavingSize), lowerLimit) < 0 ||
+ totalLeavingSize > config.getMaxSizeLeavingSource();
+ });
Review comment:
This block of code is analogous to what `canSizeEnterTarget` does for target nodes. Can we have a similar method called `canSizeLeaveSource` for source nodes?
Also, the `calculateUtilization(long plusSize)` method adds the specified plus size to used space. Here, I think we want to subtract `totalLeavingSize` from used space, instead.
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/SourceDataNodeSelectionCriteria.java
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.balancer;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.node.DatanodeUsageInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The selection criteria for selecting source datanodes , the containers of
+ * which will be moved out.
+ */
+public class SourceDataNodeSelectionCriteria {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(SourceDataNodeSelectionCriteria.class);
+ private Map<DatanodeDetails, Long> sizeLeavingNode;
+ private List<DatanodeUsageInfo> overUtilizedNodes;
+
+ public SourceDataNodeSelectionCriteria(
+ List<DatanodeUsageInfo> overUtilizedNodes,
+ Map<DatanodeDetails, Long> sizeLeavingNode) {
+ this.sizeLeavingNode = sizeLeavingNode;
+ this.overUtilizedNodes = overUtilizedNodes;
+ }
+
+ public DatanodeDetails getNextCandidateSourceDataNode() {
+ if (overUtilizedNodes.isEmpty()) {
+ LOG.info("no more candidate data node");
+ return null;
+ }
+ //TODO：use a more quick data structure, which will hava a
+ // better performance when changing or deleting one element at once
+ overUtilizedNodes.sort((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ sizeLeavingNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ sizeLeavingNode.get(b.getDatanodeDetails()));
+ //in descending order
+ return Double.compare(currentUsageOfB, currentUsageOfA);
+ });
Review comment:
My understanding of this method is unclear, so I might be wrong. I think we want to sort by reducing the used space (subtracting `sizeLeavingNode`) and then calculating utilization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-971549544
@JacksonYao287 I'll soon start working on a jira to make the default configurations smarter. Let's discuss these values and reset them in that jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-968678323
I think we should keep within threshold nodes. Removing them implies the user has to do a lot of work in first analyzing the cluster and then calculating a suitable threshold. The user can understandably expect to introduce one new node to the cluster and have balancer balance it with the default threshold.
I don't see any down side in the current logic related to within threshold nodes. There's a bug that was pointed out by
@JacksonYao287:
> in some case , one container may be moved from a withinThresholdUtilized node to another withinThresholdUtilized node
An easy fix for this is to remove `withinThresholdNodes` from `potentialTargets` when matching within threshold nodes with under utilized nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-972658644
thanks @lokeshj1703 for the review, i have updated this patch , please take a look.
> Regarding the max limits, I don't think we can support very large limits. 500 G means 500GB data can move from/to datanode in one iteration. I am not sure how much replication dn can support per minute. Maybe it should be determined first. Further we also have move timeouts, I think move would definitely time out with this much payload.
we can discuss this in @siddhantsangwan `s jira, which will make the default configurations smarter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-961761852
@lokeshj1703 @siddhantsangwan please take a look at this jira as a priority , i think it will speed up the move process very much.
by the way , i think we should not limit `maxSizeEnteringTarget` and `maxSizeLeavingSource` by default, we can specify the value in the command line if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-966943333
thanks very much @siddhantsangwan for the review, i have updated the patch , please take a look!
>By removing withinThresholdUtilizedNodes, we rely on the user setting a suitable threshold to make balancing work in such cases.
please take a look at this explanation which i left at the slack channel `ozone-balancer-discussion`
https://the-asf.slack.com/archives/C02K6DA2UNP/p1636440682032900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r751146108
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -518,14 +475,31 @@ private ContainerMoveSelection matchSourceWithTarget(
}
return null;
}
+
+ //if the utilization of the source data node becomes lower than lowerLimit
+ //after the container is moved out , then the container can not be
+ // a candidate one, and we should remove it from the candidateContainers.
+ candidateContainers.removeIf(c -> {
+ ContainerInfo cInfo;
+ try {
+ cInfo = containerManager.getContainer(c);
+ } catch (ContainerNotFoundException e) {
+ LOG.warn("Could not find container {} when " +
+ "be matched with a move target", c);
+ //remove this not found container
+ return true;
+ }
+ return !findSourceStrategy.canSizeLeaveSource(
+ source, cInfo.getUsedBytes());
+ });
Review comment:
looks good , will do this, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-972607262
Regarding the max limits, I don't think we can support very large limits. 500 G means 500GB data can move from/to datanode in one iteration. I am not sure how much replication dn can support per minute. Maybe it should be determined first.
Further we also have move timeouts, I think move would definitely time out with this much payload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752881755
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceGreedy.java
##########
@@ -49,7 +49,11 @@
double currentUsageOfB = b.calculateUtilization(
-sizeLeavingNode.get(b.getDatanodeDetails()));
//in descending order
- return Double.compare(currentUsageOfB, currentUsageOfA);
+ int ret = Double.compare(currentUsageOfB, currentUsageOfA);
+ if (ret != 0) {
+ return ret;
+ }
+ return a.hashCode() - b.hashCode();
Review comment:
Can we do datanode details UUID comparison instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752007790
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -44,37 +46,64 @@
private ContainerManager containerManager;
private PlacementPolicy placementPolicy;
+ private Map<DatanodeDetails, Long> sizeEnteringNode;
+ private NodeManager nodeManager;
+ private ContainerBalancerConfiguration config;
+ private Double upperLimit;
+ private List<DatanodeUsageInfo> potentialTargets;
public FindTargetGreedy(
ContainerManager containerManager,
- PlacementPolicy placementPolicy) {
+ PlacementPolicy placementPolicy,
+ NodeManager nodeManager) {
+ sizeEnteringNode = new HashMap<>();
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
+ this.nodeManager = nodeManager;
+ }
+
+ private void setUpperLimit(Double upperLimit){
+ this.upperLimit = upperLimit;
+ }
+
+ private void setPotentialTargets(List<DatanodeUsageInfo> potentialTargets) {
+ this.potentialTargets = potentialTargets;
+ sizeEnteringNode.clear();
+ potentialTargets.forEach(
+ p -> sizeEnteringNode.put(p.getDatanodeDetails(), 0L));
+ }
+
+ private void setConfiguration(ContainerBalancerConfiguration conf) {
+ this.config = conf;
}
/**
* Find a {@link ContainerMoveSelection} consisting of a target and
* container to move for a source datanode. Favours more under-utilized nodes.
* @param source Datanode to find a target for
- * @param potentialTargets Collection of potential target datanodes
* @param candidateContainers Set of candidate containers satisfying
* selection criteria
* {@link ContainerBalancerSelectionCriteria}
- * @param canSizeEnterTarget A functional interface whose apply
* (DatanodeDetails, Long) method returns true if the size specified in the
* second argument can enter the specified DatanodeDetails node
* @return Found target and container
*/
@Override
public ContainerMoveSelection findTargetForContainerMove(
- DatanodeDetails source, Collection<DatanodeDetails> potentialTargets,
- Set<ContainerID> candidateContainers,
- BiFunction<DatanodeDetails, Long, Boolean> canSizeEnterTarget) {
- for (DatanodeDetails target : potentialTargets) {
+ DatanodeDetails source, Set<ContainerID> candidateContainers) {
+ potentialTargets.sort((a, b) -> {
Review comment:
i think we should use TreeSet here.
in `FindSourceGreedy`, we always try to get the top one source, and if no target and be matched with it , we will remove it.
but here, we just try to find a target with Middle-order traversal。but we can not do traversal on priorityqueue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752155802
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -60,17 +61,27 @@ public FindTargetGreedy(
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
this.nodeManager = nodeManager;
+
+ potentialTargets = new TreeSet<>((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ sizeEnteringNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ sizeEnteringNode.get(b.getDatanodeDetails()));
+ return Double.compare(currentUsageOfA, currentUsageOfB);
Review comment:
good point , thanks for pointing out this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-971544876
thanks @lokeshj1703 very much for the review, i have updated the patch, please take a look!
>Max limits helps in limiting the amount of data replicated from/to dn which I think is important otherwise a dn could be overburdened with too many replications.
i do get your point，I have also considered this。but current Max limit is just the same as a default container size(5G), and i think it is too small to support moving more than one container. It depends and is debatable what value to set, so i do not specified it for now. what about 500G by default, or do you have any suggestion? we can set this in the later PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-961761852
@lokeshj1703 @siddhantsangwan please take a look at this jira as a priority , i think it will speed up the move process very much.
by the way , i think we should not limit `maxSizeEnteringTarget` and `maxSizeLeavingSource` by default, we can specify the value in the command line if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-963142634
>Maybe it is better to have a selection criteria for source datanodes as well. Default implementation would be to just return the DN with highest utilisation (also including size being moved).
thanks @lokeshj1703 for the review. the suggestion looks good , i will do this in a new commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r748058383
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -518,6 +485,27 @@ private ContainerMoveSelection matchSourceWithTarget(
}
return null;
}
+
+ //if the utilization of the source data node becomes lower than lowerLimit
+ //after the container is moved out , then the container can not be
+ // a candidate one, and we should remove it from the candidateContainers.
+ candidateContainers.removeIf(c -> {
+ ContainerInfo cInfo;
+ try {
+ cInfo = containerManager.getContainer(c);
+ } catch (ContainerNotFoundException e) {
+ LOG.warn("Could not find container {} when " +
+ "be matched with a move target", c);
+ //remove this not found container
+ return true;
+ }
+ Long totalLeavingSize = sizeLeavingNode.get(source) +
+ cInfo.getUsedBytes();
+ return Double.compare(nodeManager.getUsageInfo(source)
+ .calculateUtilization(totalLeavingSize), lowerLimit) < 0 ||
+ totalLeavingSize > config.getMaxSizeLeavingSource();
+ });
Review comment:
thanks for pointing out this, will fix it!
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -262,20 +261,17 @@ private boolean initializeIteration() {
clusterAvgUtilisation);
}
- // under utilized nodes have utilization(that is, used / capacity) less
- // than lower limit
Review comment:
sorry for this, i did not meant to remove this comment, i will add it back
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/SourceDataNodeSelectionCriteria.java
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.balancer;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.node.DatanodeUsageInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+
+/**
+ * The selection criteria for selecting source datanodes , the containers of
+ * which will be moved out.
+ */
+public class SourceDataNodeSelectionCriteria {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(SourceDataNodeSelectionCriteria.class);
+ private Map<DatanodeDetails, Long> sizeLeavingNode;
+ private List<DatanodeUsageInfo> overUtilizedNodes;
+
+ public SourceDataNodeSelectionCriteria(
+ List<DatanodeUsageInfo> overUtilizedNodes,
+ Map<DatanodeDetails, Long> sizeLeavingNode) {
+ this.sizeLeavingNode = sizeLeavingNode;
+ this.overUtilizedNodes = overUtilizedNodes;
+ }
+
+ public DatanodeDetails getNextCandidateSourceDataNode() {
+ if (overUtilizedNodes.isEmpty()) {
+ LOG.info("no more candidate data node");
+ return null;
+ }
+ //TODO：use a more quick data structure, which will hava a
+ // better performance when changing or deleting one element at once
+ overUtilizedNodes.sort((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ sizeLeavingNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ sizeLeavingNode.get(b.getDatanodeDetails()));
+ //in descending order
+ return Double.compare(currentUsageOfB, currentUsageOfA);
+ });
Review comment:
>I think we want to sort by reducing the used space (subtracting sizeLeavingNode) and then calculating utilization.
yes , that is correct. `getNextCandidateSourceDataNode` always try to return a source data node with the highest usage. thanks very much for pointing out this mistake!
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -708,8 +696,8 @@ boolean canSizeEnterTarget(DatanodeDetails target, long size) {
//2 current usage of target datanode plus sizeEnteringAfterMove
// is smaller than or equal to upperLimit
return sizeEnteringAfterMove <= config.getMaxSizeEnteringTarget() &&
- nodeManager.getUsageInfo(target)
- .calculateUtilization(sizeEnteringAfterMove) <= upperLimit;
+ Double.compare(nodeManager.getUsageInfo(target)
+ .calculateUtilization(sizeEnteringAfterMove), upperLimit) < 1;
Review comment:
the return type of `Double.compare` is int , so `< 1` is equal to `<= 0` here. but `<=0` seems more clear, will fix this, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r752155463
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
##########
@@ -60,17 +61,27 @@ public FindTargetGreedy(
this.containerManager = containerManager;
this.placementPolicy = placementPolicy;
this.nodeManager = nodeManager;
+
+ potentialTargets = new TreeSet<>((a, b) -> {
+ double currentUsageOfA = a.calculateUtilization(
+ sizeEnteringNode.get(a.getDatanodeDetails()));
+ double currentUsageOfB = b.calculateUtilization(
+ sizeEnteringNode.get(b.getDatanodeDetails()));
+ return Double.compare(currentUsageOfA, currentUsageOfB);
Review comment:
good point , thanks for pointing out this !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-971467497
Regarding the max limits - Lets consider a use case. Dn1-3 has 100TB, dn4-6 50TB and dn7-10 are new nodes. In this case all balancing would happen with dn1-3 only. Max limits helps in limiting the amount of data replicated from/to dn which I think is important otherwise a dn could be overburdened with too many replications. In the above case if we add 50 new nodes, still all balancing would happen with dn1-3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-973892519
@lokeshj1703 sure, thanks , i have merged current master branch into this patch. let`s wait for a clean CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-967032105
> How about we make the source datanodes selection criteria methods a part of the existing `ContainerBalancerSelectionCriteria` class instead of creating a new one?
looks good, so that we have only one criteria for all the selections. what do you think @lokeshj1703 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#discussion_r751123665
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
##########
@@ -341,99 +341,59 @@ private boolean initializeIteration() {
nodeManager, replicationManager, containerManager);
sourceToTargetMap = new HashMap<>(overUtilizedNodes.size() +
withinThresholdUtilizedNodes.size());
-
- // initialize maps to track how much size is leaving and entering datanodes
- sizeLeavingNode = new HashMap<>(overUtilizedNodes.size() +
- withinThresholdUtilizedNodes.size());
- overUtilizedNodes.forEach(datanodeUsageInfo -> sizeLeavingNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
- withinThresholdUtilizedNodes.forEach(datanodeUsageInfo -> sizeLeavingNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
-
- sizeEnteringNode = new HashMap<>(underUtilizedNodes.size() +
- withinThresholdUtilizedNodes.size());
- underUtilizedNodes.forEach(datanodeUsageInfo -> sizeEnteringNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
- withinThresholdUtilizedNodes.forEach(datanodeUsageInfo -> sizeEnteringNode
- .put(datanodeUsageInfo.getDatanodeDetails(), 0L));
-
return true;
}
private IterationResult doIteration() {
// note that potential and selected targets are updated in the following
// loop
- List<DatanodeDetails> potentialTargets = getPotentialTargets();
+ //TODO(jacksonyao): take withinThresholdUtilizedNodes as candidate for both
+ // source and target
+ findSourceStrategy.reInitialize(getPotentialSources(), config, lowerLimit);
+ List<DatanodeUsageInfo> potentialTargets = getPotentialTargets();
+ findTargetStrategy.reInitialize(potentialTargets, config, upperLimit);
Review comment:
Is this required for recalculating limits and other metadata in a later PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-971552516
> I'll soon start working on a jira to make the default configurations smarter. Let's discuss these values and reset them in that jira?
thanks @siddhantsangwan , sounds good, let`s determine it in that jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 edited a comment on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 edited a comment on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-966943333
thanks very much @siddhantsangwan for the review, i have updated the patch , please take a look!
>By removing withinThresholdUtilizedNodes, we rely on the user setting a suitable threshold to make balancing work in such cases.
i do get your point . i think the main difference between our ideas is what should balancer exactly do.
in your opinion, balancer should always try its best to make the cluster more balanced，no matter what the threshold is.
but in mine, balancer just try to make the cluster balanced to what the threshold specified, if we want the cluster to be more balanced, we should specified a smaller threshold.
on one hand, in practice, i think it easy to specified a smaller one if we find current threshold does not take any effect.
on the other hand, if we want the cluster to be more balanced , we can not rely on balancer`s effort with a big threshold, because how balanced balancer will make the cluster to be is uncertain. specifying a smaller threshold will definitely work as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on pull request #2808: HDDS-5517. Support multiple container moves from a source datanode in one balance iteration
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on pull request #2808:
URL: https://github.com/apache/ozone/pull/2808#issuecomment-964903287
@lokeshj1703 @siddhantsangwan @ChenSammi PTAL, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org