You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by ar...@apache.org on 2026/07/24 15:25:55 UTC
(ozone-site) branch jira2 created (now c9e97dde74)
This is an automated email from the ASF dual-hosted git repository.
ArafatKhan2198 pushed a change to branch jira2
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
at c9e97dde74 HDDS-15965. Add Recon Web UI cluster pages: Datanodes, Pipelines, Containers.
This branch includes the following new commits:
new c9e97dde74 HDDS-15965. Add Recon Web UI cluster pages: Datanodes, Pipelines, Containers.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org
(ozone-site) 01/01: HDDS-15965. Add Recon Web UI cluster pages: Datanodes, Pipelines, Containers.
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ArafatKhan2198 pushed a commit to branch jira2
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
commit c9e97dde745f3fddd6b7f9693aef4bb5ae07ed79
Author: arafat <mo...@cloudera.com>
AuthorDate: Fri Jul 24 20:55:37 2026 +0530
HDDS-15965. Add Recon Web UI cluster pages: Datanodes, Pipelines, Containers.
---
.../02-recon/01-recon-web-ui/05-datanodes.mdx | 143 ++++++++++++++++++
.../02-recon/01-recon-web-ui/06-pipelines.mdx | 115 +++++++++++++++
.../02-recon/01-recon-web-ui/07-containers.mdx | 164 +++++++++++++++++++++
.../ozone_recon_containers.png | Bin
.../ozone_recon_datanodes.png | Bin
.../ozone_recon_pipelines.png | Bin
6 files changed, 422 insertions(+)
diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/05-datanodes.mdx b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/05-datanodes.mdx
new file mode 100644
index 0000000000..fb3b3fbaa9
--- /dev/null
+++ b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/05-datanodes.mdx
@@ -0,0 +1,143 @@
+---
+sidebar_label: Datanodes
+---
+
+# Recon UI — Datanodes Page
+
+
+
+## 1. Page Overview
+
+The **Datanodes** page lists every datanode Recon knows about and shows the
+health, operational state, storage usage, pipeline membership, container counts,
+and build information for each one. It is the main place to check datanode
+health and capacity across the cluster.
+
+Besides browsing, the page lets you stop tracking datanodes that are no longer
+part of the cluster (removing dead nodes from Recon).
+
+## 2. When to Use This Page
+
+- To confirm all datanodes are healthy and in service.
+- To find stale or dead datanodes and investigate them.
+- To see which datanodes are being decommissioned or are in maintenance.
+- To check per-datanode storage usage and spot nodes that are filling up.
+- To see which pipelines a datanode participates in and how many it leads.
+- To clean up Recon's view by removing datanodes that are permanently gone.
+
+## 3. How to Access the Page
+
+Open the **Datanodes** entry in the left navigation menu, or go to the
+`/Datanodes` route directly. You can also arrive here from the **Datanodes**
+health card on the Overview page.
+
+## 4. Information Displayed
+
+The page header shows the title and an **Auto Reload** panel (see Available
+Actions). The main area is a table of datanodes, one row per node.
+
+### Table columns
+
+- **Hostname** — the datanode host (always shown, pinned to the left). Sortable.
+- **State** — datanode health, with an icon:
+ - **HEALTHY** (green check) — heartbeating normally.
+ - **STALE** (hourglass) — heartbeats delayed.
+ - **DEAD** (red cross) — not heartbeating.
+ Filterable and sortable.
+- **Operational State** — the administrative state:
+ - **IN_SERVICE** (green check) — serving normally.
+ - **DECOMMISSIONING** (hourglass) — decommission in progress.
+ - **DECOMMISSIONED** (warning) — decommission complete.
+ - **ENTERING_MAINTENANCE** (hourglass) — entering maintenance.
+ - **IN_MAINTENANCE** (warning) — under maintenance.
+ Filterable and sortable.
+- **UUID** — the datanode's unique ID. For datanodes that are actively
+ decommissioning, this cell shows a decommission summary instead of the plain
+ ID.
+- **Storage Capacity** — a bar showing used, committed, and remaining storage
+ for the node. Sortable by remaining storage.
+- **Last Heartbeat** — how long ago the node last reported in; `NA` if never.
+- **Pipeline ID(s)** — the number of pipelines the node is in; hover to see the
+ list, each with its replication type/factor and an indicator of whether this
+ node is the pipeline leader.
+- **Leader Count** — the number of Ratis pipelines in which this datanode is
+ elected leader. Sortable.
+- **Containers** — the number of containers on the node. Sortable.
+- **Open Container** — the number of open containers per pipeline. Sortable.
+- **Version** — the Ozone version running on the node. Sortable.
+- **Setup Time** — when the node was set up; `NA` if unavailable.
+- **Revision** — the build revision. Sortable.
+- **Build Date** — the build date. Sortable.
+- **Network Location** — the node's rack/network location. Sortable.
+
+## 5. Available Actions
+
+- **Columns** selector — choose which columns are visible. **Hostname** is
+ always shown.
+- **Search** — filter the shown rows by **Hostname**, **UUID**, or **Version**;
+ matching is a simple contains match on the loaded rows. Disabled when the
+ table is empty.
+- **State / Operational State filters** — filter to specific values from those
+ column headers.
+- **Sort** — click a sortable column header.
+- **Pagination** — page through results; page size is adjustable and the footer
+ shows the range and total (for example, `1-10 of 24 Datanodes`).
+- **Row selection + Remove** — select one or more datanodes to reveal a red
+ **Remove** button. Only **DEAD** datanodes can be selected; healthy or stale
+ nodes cannot. Choosing **Remove** opens a **Stop Tracking Datanode**
+ confirmation dialog before Recon stops tracking the selected node(s).
+- **Auto Reload** panel — an **Auto Refresh** toggle (refreshes every 60 seconds
+ and remembers your choice for the session; polling starts automatically on
+ this page) and a manual **reload** button showing the last refreshed time.
+- **Pipeline popover** — hover over the Pipeline ID(s) cell to see the node's
+ pipelines and leadership.
+
+## 6. How to Interpret the Information
+
+- **State HEALTHY:** the node is heartbeating normally. **STALE:** heartbeats
+ are late — the node may be under load or having network issues. **DEAD:** the
+ node has stopped heartbeating and its data is treated as unavailable.
+- **Operational State:** IN_SERVICE is the normal state. DECOMMISSIONING /
+ DECOMMISSIONED mean the node is being or has been retired. ENTERING_MAINTENANCE
+ / IN_MAINTENANCE mean it is temporarily out for maintenance.
+- **Storage Capacity bar:** watch for nodes that are nearly full or unbalanced
+ relative to others. Committed space is reserved but not yet written.
+- **Last Heartbeat:** a large value (or `NA`) alongside a STALE/DEAD state
+ confirms the node is not reporting.
+- **Leader Count:** a very uneven leader distribution across nodes can indicate
+ imbalance in Ratis leadership.
+- **Remove eligibility:** the fact that only DEAD nodes can be selected is a
+ safeguard — you cannot accidentally stop tracking a live node.
+
+## 7. Common Use Cases
+
+1. **Health sweep.** Filter **State** to STALE and DEAD to immediately see any
+ datanodes that are not healthy, then check their Last Heartbeat.
+2. **Capacity check.** Sort by **Storage Capacity** to find the fullest nodes
+ and plan rebalancing or expansion.
+3. **Clean up retired nodes.** After a node is permanently removed from the
+ cluster and shows as DEAD, select it and use **Remove** to stop Recon from
+ tracking it.
+
+## 8. Important Notes and Limitations
+
+- **Data source and freshness.** Datanode information comes from Recon's own SCM
+ view, which is built from datanode heartbeats and reports. State and storage
+ figures are only as current as the last heartbeat, and **Auto Refresh** only
+ re-queries Recon.
+- **Remove only affects Recon.** Removing a datanode stops Recon from tracking
+ it (it is removed from Recon's memory and its record in Recon's database). It
+ does not decommission or delete the node in the cluster, and only **DEAD**
+ nodes can be removed. If a removed node comes back and heartbeats, it can
+ reappear.
+- **Decommission details** for actively decommissioning nodes are shown in the
+ UUID cell as a summary.
+- **Search and filters apply to the loaded rows.**
+- On an error, the page reports a data-fetch error and the table stays empty.
+
+## 9. Related Pages
+
+- **Overview** — the Datanodes health card and cluster-wide totals.
+- **Pipelines** — details of the pipelines a datanode participates in.
+- **Containers** — container-level detail, including missing containers.
+- **Capacity** — cluster-wide storage and capacity breakdown.
diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/06-pipelines.mdx b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/06-pipelines.mdx
new file mode 100644
index 0000000000..b478a06d5d
--- /dev/null
+++ b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/06-pipelines.mdx
@@ -0,0 +1,115 @@
+---
+sidebar_label: Pipelines
+---
+
+# Recon UI — Pipelines Page
+
+
+
+## 1. Page Overview
+
+The **Pipelines** page lists the pipelines in your Ozone cluster. A pipeline is
+the group of datanodes that work together to store replicas of container data.
+For each pipeline the page shows its replication type and factor, status, the
+datanodes involved, the current leader, how long it has been alive, and
+leader-election metrics.
+
+It is a read-only listing used to review pipeline health, replication, and
+datanode membership.
+
+## 2. When to Use This Page
+
+- To see all pipelines and their current status.
+- To confirm replication type and factor across pipelines.
+- To find which datanodes make up a pipeline and which one is the leader.
+- To investigate pipelines that are not OPEN or that show frequent leader
+ elections.
+- To look up a specific pipeline by its ID.
+
+## 3. How to Access the Page
+
+Open the **Pipelines** entry in the left navigation menu, or go to the
+`/Pipelines` route directly. You can also arrive here from the **Pipelines**
+count card on the Overview page.
+
+## 4. Information Displayed
+
+The page header shows the title and an **Auto Reload** panel (see Available
+Actions). The main area is a table of pipelines, one row per pipeline.
+
+### Table columns
+
+- **Pipeline ID** — the pipeline's unique identifier (always shown, pinned to
+ the left). Sortable.
+- **Replication Type & Factor** — the replication scheme (for example RATIS) and
+ factor (for example 3), with a replication icon. Sortable.
+- **Status** — the pipeline state: OPEN, CLOSING, QUASI_CLOSED, CLOSED,
+ UNHEALTHY, INVALID, DELETED, or DORMANT. Filterable and sortable.
+- **Containers** — the number of containers in the pipeline. Sortable.
+- **Datanodes** — the datanodes that make up the pipeline, shown by hostname;
+ hover a hostname to see its UUID.
+- **Leader** — the datanode currently acting as the pipeline leader. Sortable.
+- **Last Leader Election** — elapsed time since the current leader was elected.
+ Shows `NA` unless a metrics provider such as Prometheus is configured.
+- **Lifetime** — how long the pipeline has been alive. Sortable.
+- **No. of Elections** — the number of leader elections in the pipeline. Shows
+ `NA` unless a metrics provider such as Prometheus is configured.
+
+## 5. Available Actions
+
+- **Columns** selector — choose which columns are visible. **Pipeline ID** is
+ always shown.
+- **Search** — filter the shown rows by **Pipeline ID**; matching is a simple
+ contains match on the loaded rows. Disabled when the table is empty.
+- **Status filter** — filter to specific pipeline states from the Status column
+ header.
+- **Sort** — click a sortable column header.
+- **Pagination** — page through results; page size is adjustable and the footer
+ shows the range and total (for example, `1-10 of 42 pipelines`).
+- **Auto Reload** panel — an **Auto Refresh** toggle (refreshes every 60 seconds
+ and remembers your choice for the session) and a manual **reload** button
+ showing the last refreshed time.
+- **Datanode tooltip** — hover a datanode hostname to see its UUID.
+
+## 6. How to Interpret the Information
+
+- **Status OPEN:** the pipeline is active and accepting writes. CLOSING /
+ QUASI_CLOSED / CLOSED indicate the pipeline is being or has been shut down.
+ UNHEALTHY, INVALID, DELETED, or DORMANT indicate pipelines that are not in
+ normal service and may warrant investigation.
+- **Replication Type & Factor:** confirms how many replicas the pipeline
+ maintains (for example RATIS with factor 3 keeps three copies).
+- **Datanodes / Leader:** shows exactly which nodes hold the pipeline and which
+ one leads it. Cross-reference with the Datanodes page if a member node is
+ unhealthy.
+- **Last Leader Election / No. of Elections:** frequent elections can indicate
+ instability (for example a flaky leader node or network issues). These values
+ are only populated when a metrics provider is configured; otherwise they show
+ `NA` and are not an error.
+
+## 7. Common Use Cases
+
+1. **Spot unhealthy pipelines.** Filter **Status** to values other than OPEN to
+ quickly find pipelines that are closing, unhealthy, or invalid.
+2. **Trace a pipeline's datanodes.** Search for a Pipeline ID, then read the
+ Datanodes and Leader columns to see which nodes hold its data.
+3. **Investigate leader churn.** With a metrics provider configured, sort by
+ **No. of Elections** to find pipelines with excessive leader elections and
+ correlate with datanode health.
+
+## 8. Important Notes and Limitations
+
+- **Data source and freshness.** Pipeline information comes from Recon's own SCM
+ view, built from datanode reports. It is only as current as the last refresh,
+ and **Auto Refresh** only re-queries Recon.
+- **Metrics columns need a metrics provider.** **Last Leader Election** and
+ **No. of Elections** are populated only when a metrics service such as
+ Prometheus is configured for the cluster. Without it, these columns show `NA`.
+- **Search and filters apply to the loaded rows.**
+- On an error, the page reports a data-fetch error and the table stays empty.
+
+## 9. Related Pages
+
+- **Datanodes** — health and details of the datanodes that make up pipelines.
+- **Containers** — the containers stored within pipelines.
+- **Overview** — cluster-wide totals, including the pipeline count.
diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/07-containers.mdx b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/07-containers.mdx
new file mode 100644
index 0000000000..d40e3e7a20
--- /dev/null
+++ b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/07-containers.mdx
@@ -0,0 +1,164 @@
+---
+sidebar_label: Containers
+---
+
+# Recon UI — Containers Page
+
+
+
+## 1. Page Overview
+
+The **Containers** page focuses on **unhealthy containers** — containers whose
+replicas are missing, under- or over-replicated, mis-replicated, mismatched, or
+quasi-closed. A highlights bar summarizes the counts, tabs break the containers
+down by problem type, and each container can be expanded to see the keys it
+holds. An Export tab lets you download the full list of a given unhealthy state
+as a CSV.
+
+It is the main page for diagnosing container-level replication and data-
+availability problems.
+
+## 2. When to Use This Page
+
+- To check whether any containers are missing (a data-availability risk).
+- To find under-replicated or over-replicated containers and track recovery.
+- To identify mis-replicated or replica-mismatched containers.
+- To see which keys are affected by an unhealthy container.
+- To export a full list of containers in a given state for offline analysis or
+ ticketing.
+
+## 3. How to Access the Page
+
+Open the **Containers** entry in the left navigation menu, or go to the
+`/Containers` route directly. You can also arrive here from the **Containers**
+health card on the Overview page.
+
+## 4. Information Displayed
+
+The page header shows the title and an **Auto Reload** panel (see Available
+Actions).
+
+### Highlights card
+
+A summary bar of cluster-wide container counts:
+
+- **Total Containers** — all containers in the cluster.
+- **Missing** — containers with no available replicas.
+- **Under-Replicated** — fewer replicas than expected.
+- **Over-Replicated** — more replicas than expected.
+- **Mis-Replicated** — replicas placed in a way that violates placement rules.
+- **Mismatched Replicas** — replicas that disagree (for example, differing data
+ checksums).
+- **Quasi Closed** — containers stuck in the quasi-closed state.
+
+### Tabs
+
+Each of the first six tabs shows a table of containers in that state: **Missing**,
+**Under-Replicated**, **Over-Replicated**, **Mis-Replicated**, **Mismatched
+Replicas**, and **Quasi Closed**. The seventh tab is **Export**.
+
+### Container table columns
+
+- **Container ID** — the container identifier. Sortable.
+- **No. of Blocks** — number of blocks/keys in the container. Sortable.
+- **Actual/Expected Replica(s)** — the count of replicas present versus expected
+ (for example `2 / 3`).
+- **Datanodes** — the number of datanodes holding replicas; hover to list the
+ hosts (and data checksums when available).
+- **Pipeline ID** — the pipeline the container belongs to.
+- **Unhealthy Since** — when the container entered this unhealthy state. On the
+ Quasi Closed tab this column is labeled **State Enter Time**.
+
+### Expanded row — keys in a container
+
+Clicking a container row expands it and loads the keys stored in that container,
+in a table with **Volume**, **Bucket**, **Key**, **Size**, **Date Created**,
+**Date Modified**, and **Path**.
+
+### Export tab
+
+- A **Container State** selector and an **Export CSV** button to start an export
+ job for one state.
+- **Active Exports** table (shown when jobs are running/queued): Job ID, State,
+ Status, Queue Position, Submitted, Started, and a live **Progress** bar with a
+ processed/total record count.
+- **Completed Exports** table: Job ID, State, Status, Records, Submitted,
+ Started, Completed, and an **Action** column with **Download** (showing the
+ number of downloads remaining) and **Delete**. Failed jobs show the error
+ message instead of a download button.
+
+## 5. Available Actions
+
+- **Tabs** — switch between unhealthy states; each tab loads its own data on
+ first open.
+- **Columns** selector — choose visible columns. **Container ID** is always
+ shown.
+- **Search** — filter loaded rows by **Container ID** or **Pipeline ID**
+ (contains match). Disabled when the table is empty.
+- **Sort** — click a sortable column header.
+- **Row expand** — click a row to load and view the keys in that container.
+- **Pagination** — **Previous** / **Next** buttons with a **Rows per page**
+ selector (10, 25, 50, 100). Paging uses container IDs, so it steps forward and
+ backward through the list rather than jumping to arbitrary pages.
+- **Auto Reload** panel — an **Auto Refresh** toggle (refreshes the active tab
+ and highlights every 60 seconds) and a manual **reload** button.
+- **Export actions** — start an export for a chosen state, watch progress,
+ download the resulting CSV (limited number of downloads), and delete finished
+ jobs. Only one active/completed export per state is allowed at a time; the
+ button is disabled and explains why if one already exists.
+
+## 6. How to Interpret the Information
+
+- **Missing > 0:** the most serious state — those containers have no reachable
+ replicas, so their data is currently unavailable. Investigate immediately.
+- **Under-Replicated:** fewer copies than required; the system normally
+ re-replicates automatically, but persistent entries may indicate datanode or
+ capacity problems.
+- **Over-Replicated:** more copies than required; usually self-corrects, wasting
+ some space until it does.
+- **Mis-Replicated:** replica placement violates the configured placement policy
+ (for example rack rules), even if the count is correct.
+- **Mismatched Replicas:** replicas exist but disagree — check the per-datanode
+ checksums in the Datanodes popover.
+- **Quasi Closed:** containers that could not be cleanly closed; the State Enter
+ Time shows how long they have been stuck.
+- **Actual/Expected `2 / 3`:** actual is less than expected → under-replicated;
+ greater → over-replicated.
+- **Export status:** QUEUED (waiting), RUNNING (in progress, with a progress
+ bar), COMPLETED (downloadable), FAILED (hover for the error).
+
+## 7. Common Use Cases
+
+1. **Data-availability check.** Open the page, read the **Missing** highlight,
+ and if non-zero open the Missing tab and expand rows to see exactly which
+ keys are affected.
+2. **Track replication recovery.** Watch the Under-Replicated count and tab over
+ time (with Auto Refresh on) to confirm the cluster is healing after a datanode
+ outage.
+3. **Hand off to another team.** Use the Export tab to generate a CSV of all
+ containers in a given state and attach it to a ticket for follow-up.
+
+## 8. Important Notes and Limitations
+
+- **Data source and freshness.** Unhealthy-container information comes from
+ Recon's own database, populated by Recon's periodic container-health checks;
+ the keys shown when expanding a container come from Recon's copy of the OM
+ database. Figures are only as current as those background tasks, and **Auto
+ Refresh** only re-queries Recon.
+- **The tables list unhealthy containers**, not all containers. Total container
+ count appears only in the Highlights bar.
+- **Missing count on the Overview page is capped at 1001**; this page shows the
+ detailed per-state lists.
+- **Search applies to the loaded page only** (contains match on Container ID or
+ Pipeline ID).
+- **Exports are limited:** one active/completed job per state at a time, a
+ bounded number of downloads per completed job, and a bounded queue (starting an
+ export may be rejected if the queue is full).
+- On an error, the page shows a data-fetch error; export polling errors are
+ silent by design.
+
+## 9. Related Pages
+
+- **Overview** — the Containers health card and total container count.
+- **Datanodes** — health of the datanodes that hold container replicas.
+- **Pipelines** — the pipelines containers belong to.
diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_containers.png b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_containers.png
similarity index 100%
rename from docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_containers.png
rename to docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_containers.png
diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_datanodes.png b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_datanodes.png
similarity index 100%
rename from docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_datanodes.png
rename to docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_datanodes.png
diff --git a/docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_pipelines.png b/docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_pipelines.png
similarity index 100%
rename from docs/05-administrator-guide/03-operations/09-observability/02-recon/ozone_recon_pipelines.png
rename to docs/05-administrator-guide/03-operations/09-observability/02-recon/01-recon-web-ui/ozone_recon_pipelines.png
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org