You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by we...@apache.org on 2026/09/14 04:59:06 UTC
(ozone-site) branch master updated: HDDS-16355. [Blog] Create blog for tracing. (#546)
This is an automated email from the ASF dual-hosted git repository.
jojochuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
The following commit(s) were added to refs/heads/master by this push:
new a3856df58e HDDS-16355. [Blog] Create blog for tracing. (#546)
a3856df58e is described below
commit a3856df58ef80967ba408a912de62f5c6a171da4
Author: sravani <14...@users.noreply.github.com>
AuthorDate: Mon Sep 14 10:28:59 2026 +0530
HDDS-16355. [Blog] Create blog for tracing. (#546)
---
.../hbase_flamegraph.png | Bin 0 -> 590121 bytes
.../hbase_trace.png | Bin 0 -> 477857 bytes
.../index.md | 114 +++++++++++++++++++++
.../opentelemetry_architecture.png | Bin 0 -> 1420243 bytes
.../opentelemetry_capabilities.png | Bin 0 -> 709435 bytes
blog/authors.yml | 12 +++
cspell.yaml | 3 +
7 files changed, 129 insertions(+)
diff --git a/blog/2026-09-01-distributed-tracing-opentelemetry/hbase_flamegraph.png b/blog/2026-09-01-distributed-tracing-opentelemetry/hbase_flamegraph.png
new file mode 100644
index 0000000000..27d91cfdfd
Binary files /dev/null and b/blog/2026-09-01-distributed-tracing-opentelemetry/hbase_flamegraph.png differ
diff --git a/blog/2026-09-01-distributed-tracing-opentelemetry/hbase_trace.png b/blog/2026-09-01-distributed-tracing-opentelemetry/hbase_trace.png
new file mode 100644
index 0000000000..a6dbf0ff7e
Binary files /dev/null and b/blog/2026-09-01-distributed-tracing-opentelemetry/hbase_trace.png differ
diff --git a/blog/2026-09-01-distributed-tracing-opentelemetry/index.md b/blog/2026-09-01-distributed-tracing-opentelemetry/index.md
new file mode 100644
index 0000000000..582cca00b8
--- /dev/null
+++ b/blog/2026-09-01-distributed-tracing-opentelemetry/index.md
@@ -0,0 +1,114 @@
+---
+title: "Distributed Tracing in Apache Ozone: Demystifying Performance Bottlenecks"
+authors: ["apache-ozone-community", "sravani-revuri", "sumitagrawl"]
+date: 2026-09-01
+tags: [Ozone, Distributed Tracing, OpenTelemetry, Observability]
+---
+
+If you have ever stared at a wall of logs trying to figure out why a single request took ten seconds, you know the struggle: each service tells its own story, but none of them show you the full journey.
+
+Apache Ozone is a distributed system by design. A single request can touch the Ozone Manager (OM), Storage Container Manager (SCM), and multiple Datanodes. When things slow down, metrics might tell you something is wrong but they rarely show the full path. The logs might tell you where something complained, but they're hard to interpret.
+
+Distributed tracing changes that. It maps the end-to-end path of a request with timing for every hop. Ozone now exports these paths using [OpenTelemetry](https://opentelemetry.io/) over OTLP, so you can pull them into tools like [Jaeger](https://jaegertracing.io) and actually see what happened.
+
+<!-- truncate -->
+
+## The Problem Tracing Solves
+
+Picture a slow write. Was OM stuck on metadata? SCM slow to allocate a block? Or a Datanode lagging on chunk I/O? Without tracing, you're correlating timestamps across four different log files.
+
+Ozone's stack is layered — OM for metadata, SCM for blocks, Datanodes for the actual writes — so the bottleneck could be anywhere. Tracing lays out the whole write as a single timeline so you can see exactly where it slowed down.
+
+
+
+## How It Works
+
+Ozone instruments tracing in its core services through the OpenTelemetry SDK. Trace context propagates across network boundaries, so a client request and the downstream work stay linked in one trace. Traces are exported over OTLP/gRPC (port 4317 by default) to your preferred collector, like Jaeger.
+
+
+
+## Turning It On
+
+Tracing is off by default, so the first step is enabling it in `ozone-site.xml`:
+
+```xml
+<property>
+ <name>ozone.tracing.enabled</name>
+ <value>true</value>
+</property>
+```
+
+Next, tell Ozone where to send traces. You can set `ozone.tracing.endpoint` in `ozone-site.xml`, or use the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable on each service. If you use both, the `ozone-site.xml` value takes priority.
+
+With that in place, generate some traffic — Freon is a simple option — and open Jaeger. If you see spans for services like `OzoneManager` or `freon`, tracing is working.
+
+## Keeping the Noise Down
+
+On a busy cluster, tracing every request can add overhead and flood your collector. Ozone gives you two levels of control.
+
+**Global sampling** sets the overall rate — for example, `0.01` traces 1% of requests. That's usually enough for day-to-day monitoring without much cost.
+
+**Span-level sampling** lets you go deeper on specific operations. You can always capture spans you care about, like `CommitKey` or `WriteChunk`, even when global sampling is low. That way you get broad coverage when you want it, and full detail exactly where you need it.
+
+## Tuning Without a Restart
+
+On Ozone Manager, SCM, and Datanodes, you can change tracing configs at runtime — no restart needed:
+
+- `ozone.tracing.enabled`
+- `ozone.tracing.endpoint`
+- `ozone.tracing.sampler`
+- `ozone.tracing.span.sampling`
+- `ozone.tracing.client.application-aware`
+
+That means you can flip tracing on during an incident, bump sampling to catch slow requests, then turn it back off.
+
+## Application-Aware Client Tracing
+
+Most tracing examples start inside Ozone — Freon, the shell, the S3 Gateway. In real deployments, something else is usually on top.
+
+[HBase](https://hbase.apache.org/) stores its data on Ozone through `ofs://`, and when a table creation is slow, you want the trace to start there — not halfway down the stack.
+
+Application-aware client tracing handles that. When it's on, Ozone doesn't start its own trace. It adds its work as steps inside whatever trace the application already has running.
+
+Here's an example with HBase on Ozone. Running `create 'trace_test', 'cf'` produced one trace for the whole operation — about 120ms, across HBase, Ozone Manager, SCM, and Datanodes.
+
+
+
+You can follow it from `CreateTableProcedure` on the HBase master, through the WAL sync and into Ozone's file creation and block writes.
+
+Open the step you want to see where storage time actually went — without jumping between HBase and Ozone logs trying to match things up.
+
+
+
+**To get this working:**
+
+1. [Configure HBase to store data on Ozone](https://ozone.apache.org/docs/user-guide/integrations/hbase).
+
+2. Bump up OpenTelemetry version in HBase so it can emit spans and share trace context with Ozone.
+
+3. Enable tracing on HBase following these configurations:
+
+ ```yaml
+ HBASE_OTEL_TRACING_ENABLED: "true"
+ OTEL_TRACES_EXPORTER: otlp
+ OTEL_EXPORTER_OTLP_PROTOCOL: grpc
+ OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
+ OTEL_METRICS_EXPORTER: none
+ OTEL_LOGS_EXPORTER: none
+ OTEL_TRACES_SAMPLER: always_on
+ HBASE_MASTER_OPTS: "-Dotel.resource.attributes=service.name=hbase-master"
+ HBASE_REGIONSERVER_OPTS: "-Dotel.resource.attributes=service.name=hbase-regionserver"
+ HBASE_SHELL_OPTS: "-Dotel.resource.attributes=service.name=hbase-shell"
+ ```
+
+4. On Ozone, keep `ozone.tracing.client.application-aware=true` (the default).
+
+That's it — one trace, one view, from HBase down to the Datanodes.
+
+## Why Bother?
+
+Distributed tracing pays off the first time it saves you from a multi-service log hunt. It doesn't replace metrics or Recon — it fills the gap between "this metric looks bad" and "this specific RPC on this path is why."
+
+Distributed tracing makes the invisible visible. For a system as complex as Ozone, that is the difference between debugging and guessing.
+
+If you want to go deeper, check out [HDDS-13679](https://issues.apache.org/jira/browse/HDDS-13679) and the [distributed tracing documentation](https://ozone.apache.org/docs/next/administrator-guide/operations/observability/distributed-tracing). Enable it on a dev cluster, play with the sampling, and see what your execution paths look like under the hood.
diff --git a/blog/2026-09-01-distributed-tracing-opentelemetry/opentelemetry_architecture.png b/blog/2026-09-01-distributed-tracing-opentelemetry/opentelemetry_architecture.png
new file mode 100644
index 0000000000..8240e62faf
Binary files /dev/null and b/blog/2026-09-01-distributed-tracing-opentelemetry/opentelemetry_architecture.png differ
diff --git a/blog/2026-09-01-distributed-tracing-opentelemetry/opentelemetry_capabilities.png b/blog/2026-09-01-distributed-tracing-opentelemetry/opentelemetry_capabilities.png
new file mode 100644
index 0000000000..550e2c8ec1
Binary files /dev/null and b/blog/2026-09-01-distributed-tracing-opentelemetry/opentelemetry_capabilities.png differ
diff --git a/blog/authors.yml b/blog/authors.yml
index cece1dea55..0bc52111ad 100644
--- a/blog/authors.yml
+++ b/blog/authors.yml
@@ -80,3 +80,15 @@ weimingdiit:
title: Apache Ozone Contributor
url: https://github.com/weimingdiit
image_url: https://github.com/weimingdiit.png
+
+sravani-revuri:
+ name: Sravani Revuri
+ title: Apache Ozone Contributor
+ url: https://github.com/sravani-revuri
+ image_url: https://github.com/sravani-revuri.png
+
+sumitagrawl:
+ name: Sumit Agrawal
+ title: Apache Ozone Contributor
+ url: https://github.com/sumitagrawl
+ image_url: https://github.com/sumitagrawl.png
diff --git a/cspell.yaml b/cspell.yaml
index 3c7768ae02..963d1fe989 100644
--- a/cspell.yaml
+++ b/cspell.yaml
@@ -323,6 +323,9 @@ words:
- whbing
- jianghuazhu
- weimingdiit
+- sravani
+- revuri
+- sumitagrawl
# Infrastructure and hardware terms
- firewalled
- Chrony
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org