You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vi...@apache.org on 2020/05/30 17:58:41 UTC
[hudi] branch asf-site updated: [HUDI-836] [BLOG] Monitor Hudi
metrics with Datadog (#1672)
This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ae31d77 [HUDI-836] [BLOG] Monitor Hudi metrics with Datadog (#1672)
ae31d77 is described below
commit ae31d77f16b5b78ffe387ed8a86509333f4cc15b
Author: Raymond Xu <27...@users.noreply.github.com>
AuthorDate: Sat May 30 10:58:29 2020 -0700
[HUDI-836] [BLOG] Monitor Hudi metrics with Datadog (#1672)
Add a new blog post introducing Datadog metrics reporter.
---
...0-05-28-monitoring-hudi-metrics-with-datadog.md | 65 +++++++++++++++++++++
.../blog/2020-05-28-datadog-metrics-demo.png | Bin 0 -> 105729 bytes
2 files changed, 65 insertions(+)
diff --git a/docs/_posts/2020-05-28-monitoring-hudi-metrics-with-datadog.md b/docs/_posts/2020-05-28-monitoring-hudi-metrics-with-datadog.md
new file mode 100644
index 0000000..8cc4fe4
--- /dev/null
+++ b/docs/_posts/2020-05-28-monitoring-hudi-metrics-with-datadog.md
@@ -0,0 +1,65 @@
+---
+title: "Monitor Hudi metrics with Datadog"
+excerpt: "Introducing the feature of reporting Hudi metrics via Datadog HTTP API"
+author: rxu
+category: blog
+---
+
+## Availability
+
+**0.6.0 (unreleased)**
+
+## Introduction
+
+[Datadog](https://www.datadoghq.com/) is a popular monitoring service. In the upcoming `0.6.0` release of Apache Hudi, we will introduce the feature of reporting Hudi metrics via Datadog HTTP API, in addition to the current reporter types: Graphite and JMX.
+
+## Configurations
+
+Similar to other supported reporters, turning on Datadog reporter requires these 2 properties.
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+```
+
+The following property sets the Datadog API site. It determines whether the requests will be sent to `api.datadoghq.eu` (EU) or `api.datadoghq.com` (US). Set this according to your Datadog account settings.
+
+```properties
+hoodie.metrics.datadog.api.site=EU # or US
+```
+
+The property `hoodie.metrics.datadog.api.key` allows you to set the api key directly from the configuration.
+
+```properties
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.api.key.supplier=<your api key supplier>
+```
+
+Due to security consideration in some cases, you may prefer to return the api key at runtime. To go with this approach, implement `java.util.function.Supplier<String>` and set the implementation's FQCN to `hoodie.metrics.datadog.api.key.supplier`, and make sure `hoodie.metrics.datadog.api.key` is _not_ set since it will take higher precedence.
+
+The following property helps segregate metrics by setting different prefixes for different jobs.
+
+```properties
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+Note that it will use `.` to delimit the prefix and the metric name. For example, if the prefix is set to `foo`, then `foo.` will be prepended to the metric name.
+
+There are other optional properties, which are explained in the configuration reference page.
+
+## Demo
+
+In this demo, we ran a `HoodieDeltaStreamer` job with metrics turn on and other configurations set properly.
+
+
+
+As shown above, we were able to collect Hudi's action-related metrics like
+
+- `<prefix>.<table name>.commit.totalScanTime`
+- `<prefix>.<table name>.clean.duration`
+- `<prefix>.<table name>.index.lookup.duration`
+
+as well as `HoodieDeltaStreamer`-specific metrics
+
+- `<prefix>.<table name>.deltastreamer.duration`
+- `<prefix>.<table name>.deltastreamer.hiveSyncDuration`
diff --git a/docs/assets/images/blog/2020-05-28-datadog-metrics-demo.png b/docs/assets/images/blog/2020-05-28-datadog-metrics-demo.png
new file mode 100644
index 0000000..8752ba6
Binary files /dev/null and b/docs/assets/images/blog/2020-05-28-datadog-metrics-demo.png differ