You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "errose28 (via GitHub)" <gi...@apache.org> on 2024/08/14 16:16:54 UTC
[PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
errose28 opened a new pull request, #7076:
URL: https://github.com/apache/ozone/pull/7076
## What changes were proposed in this pull request?
### Summary
In the placeholder reconciliation implementation, handling of reconcile container commands from SCM to datanodes are done in the `KeyValueHandler`. In the final implementation they should go through the `ReplicationSupervisor` like replication and reconstruction commands for proper scheduling.
This change also makes sure the same `ContainerChecksumTreeManager` instance is available in all the places it is needed within the datanode, and closes its metrics properly instead of the workaround added in [HDDS-10373](https://issues.apache.org/jira/browse/HDDS-10373).
### Specific Changes
- Usage of `ContainerChecksumTreeManager` is unified to one instance for the whole datanode.
- Merkle tree metrics object is properly closed.
- Reconcile commands are deduplicated by container ID and peer list.
- The `ReplicationSupervisor` should drop duplicate commands.
- `ReconcileContainerCommandHandler` uses `ReplicationSupervisor` for queueing. It does not implement it's own thread pool and queue.
- With this change, it also uses `ReplicationSupervisor` for metrics.
- This is similar to how `ReconstructECContainersCommandHandler` and `ReplicateContainerCommandHandler` are implemented (total time and invocation count are unused in that class)
- It would be good to have more fine grained metrics for each of these task types instead of the shared `ReplicationSupervisorMetrics`, but that would be a broader change outside the scope of this PR.
- New test helper methods for creating `KeyValueHandler` and `HddsDispatcher` instances when only a susbet of the functionality is required were added and existing tests were updated.
- This greatly reduced test dependence on the `KeyValueHandler` constructor allowing us to make changes here to get the `ContainerChecksumTreeManager` where we need it.
## What is the link to the Apache JIRA
HDDS-11254
## How was this patch tested?
- Metrics unregistration has been seen to fail existing tests if not done properly. No failures observed here.
- `TestReconcileCommandHandler` tests for metrics were removed since those are now covered by `TestReplicationSupervisor`.
- Unit test added for `TestReconcileContainerTask` that it properly detects task failure and equality.
- Existing e2e acceptance tests for reconciliation pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #7076:
URL: https://github.com/apache/ozone/pull/7076#discussion_r1717308215
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.checksum;
+
+import org.apache.hadoop.ozone.container.ozoneimpl.ContainerController;
+import org.apache.hadoop.ozone.container.replication.AbstractReplicationTask;
+import org.apache.hadoop.ozone.protocol.commands.ReconcileContainerCommand;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Objects;
+
+/**
+ * Used to execute a container reconciliation task that has been queued from the ReplicationSupervisor.
+ */
+public class ReconcileContainerTask extends AbstractReplicationTask {
+ private final ReconcileContainerCommand command;
+ private final DNContainerOperationClient dnClient;
+ private final ContainerController controller;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconcileContainerTask.class);
+
+ public ReconcileContainerTask(ContainerController controller,
+ DNContainerOperationClient dnClient, ReconcileContainerCommand command) {
+ super(command.getContainerID(), command.getDeadline(), command.getTerm());
+ this.command = command;
+ this.controller = controller;
+ this.dnClient = dnClient;
+ }
+
+ @Override
+ public void runTask() {
+ long start = Time.monotonicNow();
+
+ LOG.info("{}", this);
+
+ try {
+ controller.reconcileContainer(dnClient, command.getContainerID(), command.getPeerDatanodes());
+ setStatus(Status.DONE);
+ long elapsed = Time.monotonicNow() - start;
+ LOG.info("{} completed in {} ms", this, elapsed);
Review Comment:
We should track this as a metric as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime merged PR #7076:
URL: https://github.com/apache/ozone/pull/7076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #7076:
URL: https://github.com/apache/ozone/pull/7076#discussion_r1723836104
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/ReconcileContainerCommand.java:
##########
@@ -85,4 +87,22 @@ public String toString() {
": containerId=" + getContainerID() +
", peerNodes=" + peerDatanodes;
}
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ReconcileContainerCommand that = (ReconcileContainerCommand) o;
+ return getContainerID() == that.getContainerID() &&
+ Objects.equals(peerDatanodes, that.peerDatanodes);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(getContainerID());
Review Comment:
Yes, updated in the next commit. Default `hashCode` implementation from `AbstractSet` will compute it from all peer UUIDs which should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #7076:
URL: https://github.com/apache/ozone/pull/7076#discussion_r1723833264
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java:
##########
@@ -60,10 +60,13 @@ public class ContainerChecksumTreeManager {
public ContainerChecksumTreeManager(ConfigurationSource conf) {
fileLock = SimpleStriped.readWriteLock(
conf.getObject(DatanodeConfiguration.class).getContainerChecksumLockStripes(), true);
- // TODO: TO unregister metrics on stop.
Review Comment:
Good catch, removed in the next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #7076:
URL: https://github.com/apache/ozone/pull/7076#discussion_r1717354545
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/ReconcileContainerCommand.java:
##########
@@ -85,4 +87,22 @@ public String toString() {
": containerId=" + getContainerID() +
", peerNodes=" + peerDatanodes;
}
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ReconcileContainerCommand that = (ReconcileContainerCommand) o;
+ return getContainerID() == that.getContainerID() &&
+ Objects.equals(peerDatanodes, that.peerDatanodes);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(getContainerID());
Review Comment:
Should it include hash of the peer datanodes as well?
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.checksum;
+
+import org.apache.hadoop.ozone.container.ozoneimpl.ContainerController;
+import org.apache.hadoop.ozone.container.replication.AbstractReplicationTask;
+import org.apache.hadoop.ozone.protocol.commands.ReconcileContainerCommand;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Objects;
+
+/**
+ * Used to execute a container reconciliation task that has been queued from the ReplicationSupervisor.
+ */
+public class ReconcileContainerTask extends AbstractReplicationTask {
+ private final ReconcileContainerCommand command;
+ private final DNContainerOperationClient dnClient;
+ private final ContainerController controller;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconcileContainerTask.class);
+
+ public ReconcileContainerTask(ContainerController controller,
+ DNContainerOperationClient dnClient, ReconcileContainerCommand command) {
+ super(command.getContainerID(), command.getDeadline(), command.getTerm());
+ this.command = command;
+ this.controller = controller;
+ this.dnClient = dnClient;
+ }
+
+ @Override
+ public void runTask() {
+ long start = Time.monotonicNow();
+
+ LOG.info("{}", this);
+
+ try {
+ controller.reconcileContainer(dnClient, command.getContainerID(), command.getPeerDatanodes());
+ setStatus(Status.DONE);
+ long elapsed = Time.monotonicNow() - start;
+ LOG.info("{} completed in {} ms", this, elapsed);
Review Comment:
We should track this as a metric.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java:
##########
@@ -60,10 +60,13 @@ public class ContainerChecksumTreeManager {
public ContainerChecksumTreeManager(ConfigurationSource conf) {
fileLock = SimpleStriped.readWriteLock(
conf.getObject(DatanodeConfiguration.class).getContainerChecksumLockStripes(), true);
- // TODO: TO unregister metrics on stop.
Review Comment:
This is a related todo to clean up in `org.apache.hadoop.ozone.container.checksum.ContainerMerkleTreeMetrics#create`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #7076:
URL: https://github.com/apache/ozone/pull/7076#issuecomment-2299761843
This looks good, once the CI passes on your fork we can make it ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #7076:
URL: https://github.com/apache/ozone/pull/7076#issuecomment-2299862980
Passing on my fork [here](https://github.com/errose28/ozone/actions/runs/10478142479)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11254. Reconcile commands should be handled by datanode ReplicationSupervisor [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #7076:
URL: https://github.com/apache/ozone/pull/7076#discussion_r1723892584
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.checksum;
+
+import org.apache.hadoop.ozone.container.ozoneimpl.ContainerController;
+import org.apache.hadoop.ozone.container.replication.AbstractReplicationTask;
+import org.apache.hadoop.ozone.protocol.commands.ReconcileContainerCommand;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Objects;
+
+/**
+ * Used to execute a container reconciliation task that has been queued from the ReplicationSupervisor.
+ */
+public class ReconcileContainerTask extends AbstractReplicationTask {
+ private final ReconcileContainerCommand command;
+ private final DNContainerOperationClient dnClient;
+ private final ContainerController controller;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconcileContainerTask.class);
+
+ public ReconcileContainerTask(ContainerController controller,
+ DNContainerOperationClient dnClient, ReconcileContainerCommand command) {
+ super(command.getContainerID(), command.getDeadline(), command.getTerm());
+ this.command = command;
+ this.controller = controller;
+ this.dnClient = dnClient;
+ }
+
+ @Override
+ public void runTask() {
+ long start = Time.monotonicNow();
+
+ LOG.info("{}", this);
+
+ try {
+ controller.reconcileContainer(dnClient, command.getContainerID(), command.getPeerDatanodes());
+ setStatus(Status.DONE);
+ long elapsed = Time.monotonicNow() - start;
+ LOG.info("{} completed in {} ms", this, elapsed);
Review Comment:
Yeah we need metrics specific to reconciliation tasks since the `ReplicationSupervisorMetrics` are for all replication tasks and we aren't using the `CommandHandler` metrics anymore for anything except invocation count. I filed HDDS-11345 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org