You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2024/09/03 20:48:00 UTC
[jira] [Comment Edited] (HDDS-11341) Add dashboard for HDDS health and replication progress
[ https://issues.apache.org/jira/browse/HDDS-11341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878898#comment-17878898 ]
Ethan Rose edited comment on HDDS-11341 at 9/3/24 8:47 PM:
-----------------------------------------------------------
I'm not sure what metrics are available currently and what needs to be added. Metrics will come from SCM and datanodes. Here's a list of things it would be good to track with metrics:
* Number of datanodes in each [NodeState|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/interface-client/src/main/proto/hdds.proto#L156] (Used to track alive and dead nodes)
* Number of datanodes in each [NodeOperationalState|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/interface-client/src/main/proto/hdds.proto#L163] (Used to track decommissioning nodes)
* Number of containers in each state, similar to {{ozone admin container report}}
* Number of healthy and failed datanode storage volumes.
* Number of containers pending replication on decommissioning nodes.
* Incoming and outgoing replication traffic in bytes for each datanode.
* Number of replication and reconstruction commands queued up in on each datanode.
* Storage utilization of each datanode (might already be in another dashboard as well)
We could add container balancer metrics to this dashboard as well, but that might be too cluttered. It might be better to leave balancer in a separate dashboard since that is run on demand vs the items identified here which happen in response to failures.
was (Author: erose):
I'm not sure what metrics are available currently and what needs to be added. Metrics will come from SCM and datanodes. Here's a list of things it would be good to track with metrics:
* Number of datanodes in each [NodeState|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/interface-client/src/main/proto/hdds.proto#L156] (Used to track alive and dead nodes)
* Number of datanodes in each [NodeOperationalState|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/interface-client/src/main/proto/hdds.proto#L163] (Used to track decommissioning nodes)
* Number of containers in each state, similar to {{ozone admin container report}}
* Number of healthy and failed datanode storage volumes.
* Number of containers pending replication on decommissioning nodes.
* Incoming and outgoing replication traffic in bytes for each datanode.
* Number of replication and reconstruction commands queued up in on each datanode.
> Add dashboard for HDDS health and replication progress
> ------------------------------------------------------
>
> Key: HDDS-11341
> URL: https://issues.apache.org/jira/browse/HDDS-11341
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Dashboards
> Reporter: Ethan Rose
> Priority: Major
>
> Add a Grafana dashboard to show information about datanode health, ongoing and pending replication and reconstruction tasks, and the amount of data being moved between nodes due to these tasks. This board will be useful to monitor during disk failure, node failure, node decom, and maintenance.
> SCM replication manager likely has a lot of the metrics for ongoing tasks already. We may need to add more metrics to datanodes to monitor tasks that are ongoing (not just those that are queued) and the amount of data being moved. I think some datanode command queue and handler related metrics are unused as well and those can be checked/removed/updated as part of this PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org