You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/06/28 00:56:52 UTC
[GitHub] [ozone] kerneltime opened a new pull request, #3562: HDDS-6959. Fix move timeout in latest iteration metric
kerneltime opened a new pull request, #3562:
URL: https://github.com/apache/ozone/pull/3562
## What changes were proposed in this pull request?
Fix move timeout in latest iteration metric
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6959
## How was this patch tested?
Added a basic unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3562: HDDS-6959. Fix move timeout in latest iteration metric
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3562:
URL: https://github.com/apache/ozone/pull/3562#issuecomment-1168810918
> @kerneltime Thanks for working on this. I think we need some refactoring here. With this change, post move book-keeping will be done at three different places:
>
> 1. `whenComplete((result, ex) -> {...}` in `ContainerBalancer#moveContainer`
This gets called when the underlying replication manager completes
> 2. `ContainerBalancer#checkIterationMoveResults`
This gets evaluated for timeout when the logic in ContainerBalancer gives up on the replication.
> 3. `ContainerBalancerMetrics#incrementCurrentIterationContainerMoveMetric` which is called in `ContainerBalancer#moveContainer`
This is the same as 1.
>
> Do you think it's better to update metrics at just one place, preferably `ContainerBalancer#checkIterationMoveResults`?
There are 2 different ways by which a timeout event can occur. We could split the timeout into 2 separate counters based on who gave up but that should be a different PR.
There are 2 tests in `TestContainerBalancer` which trigger different ways in which timeout can occur and increment the same counter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #3562: HDDS-6959. Fix move timeout in latest iteration metric
Posted by GitBox <gi...@apache.org>.
adoroszlai merged PR #3562:
URL: https://github.com/apache/ozone/pull/3562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] siddhantsangwan commented on a diff in pull request #3562: HDDS-6959. Fix move timeout in latest iteration metric
Posted by GitBox <gi...@apache.org>.
siddhantsangwan commented on code in PR #3562:
URL: https://github.com/apache/ozone/pull/3562#discussion_r908489943
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java:
##########
@@ -121,6 +122,42 @@ public void incrementNumContainerMovesCompletedInLatestIteration(
this.numContainerMovesCompletedInLatestIteration.incr(valueToAdd);
}
+ public void incrementCurrentIterationContainerMoveMetric(
+ MoveResult result,
+ long valueToAdd) {
+ if (result == null) {
+ return;
+ }
+ switch (result) {
+ case COMPLETED:
+ this.numContainerMovesCompletedInLatestIteration.incr(valueToAdd);
+ break;
+ case REPLICATION_FAIL_TIME_OUT:
+ this.numContainerMovesTimeoutInLatestIteration.incr(valueToAdd);
+ break;
+ // TODO: Add metrics for other errors that need to be tracked.
+ case FAIL_NOT_RUNNING:
+ case REPLICATION_FAIL_INFLIGHT_REPLICATION:
+ case FAIL_NOT_LEADER:
+ case REPLICATION_FAIL_NOT_EXIST_IN_SOURCE:
+ case REPLICATION_FAIL_EXIST_IN_TARGET:
+ case REPLICATION_FAIL_CONTAINER_NOT_CLOSED:
+ case REPLICATION_FAIL_INFLIGHT_DELETION:
+ case REPLICATION_FAIL_NODE_NOT_IN_SERVICE:
+ case DELETION_FAIL_NODE_NOT_IN_SERVICE:
+ case REPLICATION_FAIL_NODE_UNHEALTHY:
+ case DELETION_FAIL_TIME_OUT:
Review Comment:
Timeouts when trying to delete container from source DN should also be added to the `numContainerMovesTimeoutInLatestIteration` metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3562: HDDS-6959. Fix move timeout in latest iteration metric
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3562:
URL: https://github.com/apache/ozone/pull/3562#issuecomment-1171581057
Thanks @kerneltime for the fix, @siddhantsangwan for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3562: HDDS-6959. Fix move timeout in latest iteration metric
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3562:
URL: https://github.com/apache/ozone/pull/3562#discussion_r908588274
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java:
##########
@@ -121,6 +122,42 @@ public void incrementNumContainerMovesCompletedInLatestIteration(
this.numContainerMovesCompletedInLatestIteration.incr(valueToAdd);
}
+ public void incrementCurrentIterationContainerMoveMetric(
+ MoveResult result,
+ long valueToAdd) {
+ if (result == null) {
+ return;
+ }
+ switch (result) {
+ case COMPLETED:
+ this.numContainerMovesCompletedInLatestIteration.incr(valueToAdd);
+ break;
+ case REPLICATION_FAIL_TIME_OUT:
+ this.numContainerMovesTimeoutInLatestIteration.incr(valueToAdd);
+ break;
+ // TODO: Add metrics for other errors that need to be tracked.
+ case FAIL_NOT_RUNNING:
+ case REPLICATION_FAIL_INFLIGHT_REPLICATION:
+ case FAIL_NOT_LEADER:
+ case REPLICATION_FAIL_NOT_EXIST_IN_SOURCE:
+ case REPLICATION_FAIL_EXIST_IN_TARGET:
+ case REPLICATION_FAIL_CONTAINER_NOT_CLOSED:
+ case REPLICATION_FAIL_INFLIGHT_DELETION:
+ case REPLICATION_FAIL_NODE_NOT_IN_SERVICE:
+ case DELETION_FAIL_NODE_NOT_IN_SERVICE:
+ case REPLICATION_FAIL_NODE_UNHEALTHY:
+ case DELETION_FAIL_TIME_OUT:
Review Comment:
Makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org