You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2022/10/26 10:36:35 UTC
[GitHub] [incubator-pegasus] empiredan opened a new issue, #1206: Featureļ¼provide a RESTful service to query immediate values of metrics from new framework
empiredan opened a new issue, #1206:
URL: https://github.com/apache/incubator-pegasus/issues/1206
# 1. Motivation
The new framework should provide a service to collect metrics. Since it's hoped that the metrics are collected very conveniently, the service can be **RESTful**. With *http* protocol, the metrics can be viewed just by any browser or a `curl` command.
# 2. Data types
## 2.1 Request
Common RESTful semantics will be adopted for the service. For example, metrics can queried by `GET` method with parameters passed in [query string](https://en.wikipedia.org/wiki/Query_string).
However, there are just field pairs (name and value) in query string. In our scenario, complex data structures such as array should be supported. For example, multiple metric names may be contained in a query.
To solve this problem, the field value can be designed as such format to represent an array: comma("`,`") can be used as the delimiter to separate each element of the array, for example "`put_count,put_latency,alive_node_count`"; also, comma("`,`") can be used safely in query component in URL according to [possible side effect using comma in query string](https://stackoverflow.com/questions/45686595/possible-side-effect-using-comma-in-querystring) and [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986#section-3.4).
## 2.2 Response
The collected metrics can just be organized in *json* format. The detailed implementations will be described in the following sections.
# 3. API
## 3.1 Query metrics
### 3.1.1 Request
#### URI
```url
/metrics
```
#### Parameters
| Parameters | Data Types |Description|
|:----------:|:-----------:|:----------|
| types | array | entity types, such as `server`, `table`, `replica`, etc. |
| ids | array | entity IDs. |
| attributes | array | attributes for entity, such as table name and partition id for entity `replica`, and "`attr_key_1,attr_val_1,attr_key_2,attr_val_2`" means there are 2 pairs of attributes: `attr_key_1:attr_val_1` and `attr_key_2:attr_val_2`. |
| metrics | array | metric names. |
#### Headers
```http
Accept: application/json
```
### 3.1.2 Response
#### Headers
```http
Content-Type: application/json
```
#### Content
```json
[
{ // entity 1
"type": "<entity_type>",
"id": "<entity_id>",
"attributes": {
"<attr_key_1>": "<attr_val_1>",
"<attr_key_2>": "<attr_val_2>",
"<attr_key_3>": "<attr_val_3>",
...
},
"metrics": [ // metrics that belong to entity 1
// gauge
{
"name": "<gauge_name>",
"value": 100
},
// counter
{
"name": "<counter_name>",
"value": 10000
},
// percentile
{
"name": "<percentile_name>",
"p50": 1,
"90": 5,
"95": 8,
"99": 10,
"p999": 15
}
]
},
{ // entity 2
...
},
...
]
```
### 3.1.3 Examples
#### Collect all metrics
```url
/metrics
```
#### Collect all metrics from server entity
```url
/metrics?types=server
```
#### Collect all metrics from both server and replica entity
```url
/metrics?types=server,replica
```
#### Collect all metrics of the given replica entities (table_name=my_table, partitions_id=1 or 5)
```url
/metrics?types=replica&attributes=table_name,my_table,partition_id,1,partition_id,5
```
#### Collect specified metrics of server entity and the given replica entities (table_name=my_table, partitions_id=1 or 5)
```url
/metrics?types=server,replica&attributes=table_name,my_table,partition_id,1,partition_id,5,metrics=get_latency,put_count,alive_node_count
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] empiredan closed issue #1206: Feature(new_metrics): provide a RESTful service to query immediate values of metrics from new framework
Posted by "empiredan (via GitHub)" <gi...@apache.org>.
empiredan closed issue #1206: Feature(new_metrics): provide a RESTful service to query immediate values of metrics from new framework
URL: https://github.com/apache/incubator-pegasus/issues/1206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] empiredan commented on issue #1206: Feature(new_metrics): provide a RESTful service to query immediate values of metrics from new framework
Posted by GitBox <gi...@apache.org>.
empiredan commented on issue #1206:
URL: https://github.com/apache/incubator-pegasus/issues/1206#issuecomment-1298191783
> Some questions:
>
> 1. In the response, I know the metrics will be filtered out, but will all the `type`, `id` and `attributes` (with all attribute k-v pairs) be returned no matter what the query string is ?
> 2. Is it helpful to return the description of each metrics, such as `statistic the qps of PUT request` for "put_qps" ? We can add a more parameter to decide whether to return it, because it's verbos if return it every time.
> 3. Metrics type (e.g. counter, guage, percentile) is same to point 2.
Good questions ! I'll answer each as follows:
1. Yes, in current design, all of `type`, `id` and `attributes` with all k-v pairs will be returned. And I think returned fields of entities can be custom.
2. Yes, actually the description for metric has been a property of `metric_prototype`, and certainly can be a field returned. As is put in the answer for 1st question, to support custom fields, 2 parameters can be introduced: `with_entity_field` and `without_entity_field`. For example, `with_entity_field=type,desc` will only call for `type` and `desc` fields; and `without_entity_field=id,desc` will return all fields except `id` and `desc`, is this alright ?
3. As for metric, use `with_metric_field` and `without_metric_field`. For example, `with_metric_field=value,p99` means it will just return the `value` of gauge or counter, and `p99` of percentile, if any.
4. Just use the parameters in answer for 3rd question, percentile types can also be custom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on issue #1206: Feature(new_metrics): provide a RESTful service to query immediate values of metrics from new framework
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #1206:
URL: https://github.com/apache/incubator-pegasus/issues/1206#issuecomment-1297991963
4. Return all percentiles for percentile type metrics is fast to implement, will you design some type of filter for it in the future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on issue #1206: Feature(new_metrics): provide a RESTful service to query immediate values of metrics from new framework
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #1206:
URL: https://github.com/apache/incubator-pegasus/issues/1206#issuecomment-1298285235
Good, let's go ahead!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on issue #1206: Feature(new_metrics): provide a RESTful service to query immediate values of metrics from new framework
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #1206:
URL: https://github.com/apache/incubator-pegasus/issues/1206#issuecomment-1297989084
Some questions:
1. In the response, I know the metrics will be filtered out, but will all the `type`, `id` and `attributes` (with all attribute k-v pairs) be returned no matter what the query string is ?
2. Is it helpful to return the description of each metrics, such as `statistic the qps of PUT request` for "put_qps" ? We can add a more parameter to decide whether to return it, because it's verbos if return it every time.
3. Metrics type (e.g. counter, guage, percentile) is same to point 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org