You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2022/03/04 10:51:50 UTC
[GitHub] [incubator-pegasus] empiredan opened a new issue #922: Feature: implement the framework of the new metrics system
empiredan opened a new issue #922:
URL: https://github.com/apache/incubator-pegasus/issues/922
# 1. Motivation
The root cause of introducing a new metrics system is the unreasonable naming of the perf-counter, as it's been decribed in [2020-08-27-metric-api.md](https://github.com/apache/incubator-pegasus/blob/master/rfcs/2020-08-27-metric-api.md). The naming is verbose: besides that there are meaningless words in it, some terms can be moved as the labels/tags which tend to have common properties.
There're also other reasons to refactor the perf-counter.
Firstly, in perf-counter, the types of metrics have not been strictly separated, such as gauge and counter. For example, *set* is not an efficient operation for counter.
Secondly, as the base class, perf-counter has all the abstract interfaces for each type of metric. Every sub class has to implement all the abstract interfaces. For example, it is unreasonable that a gauge has *get_percentile* method, though it can simply be implemented as `dassert(false, ...);`.
Thirdly, there're performance problems for `perf_counter_number_atomic`. It should be optimized and a more efficient counter should be provided.
# 2. The Framework
Even if we need to implement a new framework, an alternative solution is that we can turn to mature and active metrics lib for c++. However, there's few of this kind of libs. https://github.com/jupp0r/prometheus-cpp is a possible candidate. It implements [Prometheus Data Model](https://prometheus.io/docs/concepts/data_model/) which is natually supported by Prometheus. It can also be exported and translated into the other data model of metrics. Nevertheless, its implementation is not efficient enough for high-concurency system; even sampling for [Summary](https://prometheus.io/docs/concepts/metric_types/#summary) (a metric type of Prometheus, also known as percentile) will block the task thread.
Therefore, just like what we've done before, we should introduce an abstraction level for the new metrics system with well-defined metric types; then, collect the metric data and translate them into the metric models supported by each specific monitoring system, such as Prometheus or Open Falcon; finally, the metric data are gathered by the monitoring system and can be observed very conveniently.
There are some requirements for the new abstraction:
* It should be able to applied to all of the existing metrics;
* It should support to manage both of common and exclusive properties of the metrics, which later may be used as labels/tags;
* It should provide separate and well-defined metric types;
* The provided metric sampling should be fast enough without excessive cpu and memory consumption, and never block the task thread.
The new abstraction level is inspired by [Kudu Metrics](https://github.com/apache/kudu/blame/master/src/kudu/util/metrics.h). Next I'll describe how it is implemented.
## 2.1 Metric Registery
All of the metrics are maintained in the metric registry. The metric registery is a singleton. While a metric entity (see 2.2) is instantiated, it will be registered into the registry.
The main use of registry is to collect all of metric data and export to the specific data sink (see 2.5), since it knows every metric.
## 2.2 Metric Entity Prototype
Each metric has a corresponding *level*, which is called an *entity*, such as the memory of a **server**, the put latency of a **table**, the get latency of a **replica**. All of **server**, **table** and **replica** are entities.
`METRIC_DEFINE_entity(...)` macro can be used to define an entity prototype. An entity prototype is an abstract class of an entity. To become a concrete entity, a prototype should be instantiated with some attributions. As is mentioned in 2.1, during instantiation the entity will also be registered into the registry with its unique ID.
For example, a **server** can be instantiated with its IP or hostname; a **table** can be instantiated with its app id or name; a **replica** can be instantiated with its app id and partition id.
## 2.3 Metric Prototype
Similar with the entity prototype, a metric should also be instantiated from a metric prototype. A metric prototype defines the basic meta information of a metric, including the entity it is attached to, its name, its unit (such as bytes, milliseconds, operations per second, etc.), and its description. To define a metric prototype, just use `METRIC_DEFINE_*` macro.
Once a metric prototype is defined, it can be used to instantiate a metric, which will also be attached to the specific entity thus maintained by a registry.
## 2.4 The Types of Metrics
According to the current metrics we've used in Pegasus, there are 5 types for the new framework: gauge, counter, volatile counter, meter and percentile. For simplicity，a brief introduction is given for each type; the implementation details will be described in the each issue of the types.
### 2.4.1 The Gauge
A gauge is a point-in-time measurement. The value got from a gauge is exactly the same as what it has been recently set.
There are 2 primitive types supported for the gauge: `int64_t` for a value of integer and `double` for a value of floating-point.
An example of a gauge is `meta*eon.meta_service*unalive_nodes`, which shows the current number of dead replica servers.
### 2.4.2 The Counter
Typically a counter is a metric that only monotonically increases ([Counter](https://prometheus.io/docs/concepts/metric_types/#counter) in Prometheus). Similarly, in pegasus, there are metrics of this kind of counter, such as `zion*profiler*RPC_RRDB_RRDB_PUT.cancelled`.
However, in Pegasus there are also another kind of counter that may sometimes decrease, such as `replica*app.pegasus*manual.compact.running.count`, which will increase while a new manual compact is started, and decrease while it is finished.
Therefore, the counter implemented for Pegasus should support both increase and decrease, just like what has been done in [Counters](https://metrics.dropwizard.io/4.2.0/manual/core.html#man-core-counters) of Dropwizard.
### 2.4.3 The Volatile Counter
Fetching value from a general counter is trivial: just return the current value of the counter is ok.
However, many metrics are "recent" in Pegasus, which means that the historically-accumulated count is ignored. If a counter is "recent", it will be reset to 0 immediately after its value is fetched. Thus the "recent" is the duration between 2 succesive accesses to the counter. For example `replica*eon.failure_detector*recent_beacon_fail_count` means how many failed beacons there are recently.
According to this scenario, a special counter (called a volatile counter) is implemented. The only difference between it and a general counter is that once the value is read, a volatile counter will be immediately cleared to 0, while a general counter is kept unchanged.
### 2.4.4 The Meter
A meter measures the rate of occurrences of a set of events over time. In Pegasus the typical application is QPS, such as `zion*profiler*RPC_RRDB_RRDB_GET.qps`, which measures the rate at which a replica server reads a single value.
It should be noted that the underlying counter of a meter is volatile, which means it will be reset to 0 immediately after the value of the meter is fetched.
### 2.4.5 The Percentile
Like [Summary](https://prometheus.io/docs/concepts/metric_types/#summary) in Prometheus, the metric type of percentile samples observations. Periodically it calculates configurable percentiles (p50, p75, p90, p95, p99, p999, etc.) over the recent samples.
The most common usage of the percentile is latency, such as `zion*profiler*RPC_RRDB_RRDB_PUT.latency.server.p999`, which is the 999-th percentile of server-side latency between the moment the task is pushed into the queue and the point the server begins to reply to client.
## 2.5 Metric Data Sink
As it is described in the beginnig of this chapter, the ultimate goal is to show the metrics in the monitoring system. Therefore, the metric data sink, as its name implies, actually is the metric model supported by each monitoring system which the metric data are collected to.
Based on this design, the base class of the data sink must be abstract. It is necessary to implement the sub class for each metric model of target, such as Prometheus or Open Falcon.
In the sub class, the metric data will be translated into the specific model. For Prometheus, a client should be created and initialized, listening on a port which is used for the Prometheus server to pull data.
# 3. Examples
## 3.1 Define & Instantiate Metric Entities
In this section, some examples are given to show how to define and instantiate metric entities.
### 3.1.1 Define & Instantiate Server Entity
```c++
// Define server entity
METRIC_DEFINE_entity(server);
// Instantiate server entity
ref_ptr<metric_entity> get_metric_entity_server(const char *ip_addr)
{
static auto server_entity = METRIC_ENTITY_server.instantiate(
"server",
{{"ip", ip_addr}}
);
return server_entity;
}
```
### 3.1.2 Define & Instantiate Table Entity
```c++
// Define table entity
METRIC_DEFINE_entity(table);
// Instantiate table entity
ref_ptr<metric_entity> get_metric_entity_table(const char *table_name)
{
static auto table_entity = METRIC_ENTITY_table.instantiate(
table_name,
{{"table", table_name}}
);
return table_entity;
}
```
### 3.1.3 Define & Instantiate Replica Entity
```c++
// Define replica entity
METRIC_DEFINE_entity(replica);
// Instantiate replica entity
ref_ptr<metric_entity> get_metric_entity_replica(const char *table_name, int32_t partition_id)
{
auto replica_id = fmt::format("{}:{}", table_name, partition_id);
static auto replica_entity = METRIC_ENTITY_replica.instantiate(
table_name,
{{"table", table_name}, {"partition", std::to_string(partition_id)}}
);
return replica_entity;
}
```
## 3.2 Define metrics
In this section, take the put latency for example to show how to define and instantiate metrics.
### 3.2.1 Define & Instantiate Server-level Metric
```c++
// Define server-level metric
METRIC_DEFINE_percentile(server, server_put_latency, kNanoSeconds, "the server-level latency of put requests");
// Instantiate metric
auto server_put_latency = METRIC_server_put_latency.instantiate(
get_metric_entity_server("10.0.0.1"),
{90, 95, 99}
);
```
### 3.2.2 Define & Instantiate Table-level Metric
```c++
// Define replica-level metric
METRIC_DEFINE_percentile(table, table_put_latency, kNanoSeconds, "the table-level latency of put requests");
// Instantiate metric
auto table_get_latency = METRIC_table_put_latency.instantiate(
get_metric_entity_table("test_app1"),
{90, 99}
);
```
### 3.2.3 Define & Instantiate Replica-level Metric
```c++
// Define replica-level metric
METRIC_DEFINE_percentile(replica, replica_put_latency, kNanoSeconds, "the replica-level latency of put requests");
// Instantiate metric
auto replica_get_latency = METRIC_replica_put_latency.instantiate(
get_metric_entity_replica("test_app1", "2"),
{50, 75, 90, 95, 99, 999}
);
```
# 4. Schedule
All of the sub tasks to implement the new framework are listed here to track the status of each of them.
- [x] https://github.com/apache/incubator-pegasus/issues/889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org