You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2021/02/22 06:38:21 UTC
[GitHub] [couchdb] tonysun83 opened a new issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
tonysun83 opened a new issue #3377:
URL: https://github.com/apache/couchdb/issues/3377
---
name: Formal RFC
about: Submit a formal Request For Comments for consideration by the team.
title: Support for native Prometheus Endpoints
labels: rfc, discussion
assignees: @tonysun83
---
[NOTE]: # ( ^^ Provide a general summary of the RFC in the title above. ^^ )
# Introduction
This is a formal proposal to add a `/_node/{node-name}/_metrics` endpoint that outputs https://prometheus.io/ metrics data. @garrensmith and @jiangphcn began this discussion in the mailing list and this proposal consolidates the list of options for this new endpoint.
## Abstract
Currently, CouchDB's metrics and diagnostic information can be obtained via node specific`_stats`, `_active_tasks`, and `_system` endpoints. Prometheus has become the more prevalent and standard approach for exposing metrics. Adding support to expose CouchDB metrics in Prometheus format is something to be desired as demonstrated by [CouchDB Prometheus Exporter](https://github.com/gesellix/couchdb-prometheus-exporter).
One solution is to bundle the [CouchDB Prometheus Exporter](https://github.com/gesellix/couchdb-prometheus-exporter) as part of CouchDB. This requires bundling GO as part of the build and also does not include `_system` info.
The proposed solution is to add a native module or app that receives the `/_node/{node-name}/_metrics` call and consolidates the `_stats`, `_active_tasks`, and `_system` calls into one response. The default format will be JSON and an option will be provided for Prometheus.
## Requirements Language
[NOTE]: # ( Do not alter the section below. Follow its instructions. )
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in
[RFC 2119](https://www.rfc-editor.org/rfc/rfc2119.txt).
## Terminology
TBD - To Be Determined
[TIP]: # ( Provide a list of any unique terms or acronyms, and their definitions here.)
---
# Detailed Description
1) A new module, or perhaps a new app will be written to consolidate the results of our `_stats`, `active_task`, and `_system` internal function calls. The exact form is TBD. The JSON format should be straightforward since it is just aggregation of existing calls. However the Prometheus format requires more detailed implementation for conversion. See the [Http API](#HTTPAPI) for return format proposals.
2) The endpoint will be node specific and not for the entire cluster. It's up to an external monitoring tool to aggregate and present the entire cluster's data. This the current design choice and is open for discussion. The exact endpoint is:
`/_node/{node-name}/_metrics`. The endpoint will have an optional Accept Header that determines whether JSON or Prometheus output is returned.
# Advantages and Disadvantages
**Advantages**
- Native functionality without relying on external converters.
- No need to bundle GO as part of our release (we would need to bundle GO if we simply included [CouchDB Prometheus Exporter](https://github.com/gesellix/couchdb-prometheus-exporter))
- Ability to see all metrics info via one endpoint
**Disadvantages**
- Prometheus scraping would require issuing a `_metrics` endpoint call for every node.
- Re-implementation of functionality (_stats, _active_tasks) already available with [CouchDB Prometheus Exporter](https://github.com/gesellix/couchdb-prometheus-exporter)
- A standard that is obsolete in the future.
# Key Changes
A new node specific endpoint will be added to the API.
## Applications and Modules affected
1) chttpd
- chttpd_node
- chttpd_prometheus (new module)
2) <couchdb-prometheus> (new app perhaps)
<a name="HTTPAPI"></a>
## HTTP API additions
**GET** `/_node/{node-name}/_metrics` HTTP/1.1
Returns consolidated metrics info (_stats, _active_tasks, _system) via JSON or Prometheus standard.
**Request Headers: Headers:**
If no header is provided, the default is JSON. Content-Type application/json will return JSON, while prometheus will return prometheus formatting.
Accept:
- application/json
- prometheus
**Response Headers:**
Content-Type:
- application/json
- text/plain; charset=utf-8
Valid Status Codes
200 OK - Request completed successfully
JSON Response Example
```
[{
"value": {
"min": 0,
"max": 0,
"arithmetic_mean": 0,
"geometric_mean": 0,
"harmonic_mean": 0,
"median": 0,
"variance": 0,
"standard_deviation": 0,
"percentile": [
[
50,
0
],
[
75,
0
],
[
90,
0
],
[
95,
0
],
[
99,
0
],
[
999,
0
]
],
"histogram": [
[
0,
0
]
]
}
},
{
"node": "node1@127.0.0.1",
"pid": "<0.622.0>",
"changes_done": 199,
"current_version_stamp": "8131141649532-0-198",
"database": "testdb",
"db_version_stamp": "8131141649532-0-999",
"design_document": "_design/example",
"started_on": 1594703583,
"type": "indexer",
"updated_on": 1594703586
},
{
"uptime": 259,
"memory": {
...
}
}
]
```
Sample Prometheus Output:
```
# TYPE couchdb_uptime_seconds counter
couchdb_uptime_seconds 1
# TYPE couchdb_erlang_memory_bytes gauge
couchdb_erlang_memory_bytes{memory_type="total"} 71237784
couchdb_erlang_memory_bytes{memory_type="processes"} 12248504
couchdb_erlang_memory_bytes{memory_type="processes_used"} 12235928
couchdb_erlang_memory_bytes{memory_type="system"} 58989280
couchdb_erlang_memory_bytes{memory_type="atom"} 1172689
couchdb_erlang_memory_bytes{memory_type="atom_used"} 1156575
couchdb_erlang_memory_bytes{memory_type="binary"} 182568
couchdb_erlang_memory_bytes{memory_type="code"} 27819083
couchdb_erlang_memory_bytes{memory_type="ets"} 3143536
# TYPE couchdb_erlang_gc_collections_total counter
couchdb_erlang_gc_collections_total 13417
# TYPE couchdb_erlang_gc_words_reclaimed_total counter
couchdb_erlang_gc_words_reclaimed_total 71296018
# TYPE couchdb_erlang_context_switches_total counter
couchdb_erlang_context_switches_total 358276
# TYPE couchdb_erlang_reductions_total counter
couchdb_erlang_reductions_total 46527253
# TYPE couchdb_erlang_processes gauge
couchdb_erlang_processes 528
# TYPE couchdb_erlang_process_limit gauge
couchdb_erlang_process_limit 262144
# TYPE couchdb_erlang_io_recv_bytes_total counter
couchdb_erlang_io_recv_bytes_total 23291839
# TYPE couchdb_erlang_io_sent_bytes_total counter
couchdb_erlang_io_sent_bytes_total 8915261
# TYPE couchdb_erlang_message_queues gauge
couchdb_erlang_message_queues 0
# TYPE couchdb_erlang_message_queue_min gauge
couchdb_active_task{type="replication", source="mailbox", target="http://mailsrv:5984/mailbox <http://mailsrv:5984/mailbox>", docs_count = "docs_read"} 4524
couchdb_active_task{type="replication", source="mailbox", target="http://mailsrv:5984/mailbox <http://mailsrv:5984/mailbox>", docs_count = "docs_written"} 4524
couchdb_active_task{type="replication", source="mailbox", target="http://mailsrv:5984/mailbox <http://mailsrv:5984/mailbox>", docs_count = "missing_revisions_found"} 4524
```
## HTTP API deprecations
None
# Security Considerations
N/A
# References
http://couchdb-development.1959287.n2.nabble.com/DISCUSS-Prometheus-endpoint-in-CouchDB-4-x-td7607648.html
# Acknowledgements
Summary and implementation ideas mostly from the mailing list discussion responses.
@jiangphcn @garrensmith @janl @wohali @davisp @willholley
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva closed issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
nickva closed issue #3377:
URL: https://github.com/apache/couchdb/issues/3377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rnewson commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
rnewson commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-789206650
looks good to me, direct and to the point.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] kocolosk commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
kocolosk commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-961355827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-961334939
Prometheus support shipped in 3.2.0
https://docs.couchdb.org/en/3.2.0/whatsnew/3.2.html#features-and-enhancements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] kocolosk commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
kocolosk commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-961355827
Yeah, I was just wondering if there's supposed to be some place where we store these RFCs in perpetuity. We started using the couchdb-documentation repo for this purpose but that didn't really feel natural either.
Agree this issue can be closed, but I think there's an opportunity for improvement in the RFC process here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rnewson commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
rnewson commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-783250157
It seems the discussion is still open, are we to continue it here in the Issue or on the mailing list? I was about to raise several points but they've all come up in the discussion (without resolution).
Consolidating my thoughts;
1) We're adding support for Prometheus but also inventing a new format for it that nothing understands.
2) Negotiation by content-type when Prometheus doesn't have a registered one is a mistake (The accept= query parameter idea is better).
3) As not every item in active_tasks is a metric, the new endpoint is not a replacement for it, so we'll have to retain both.
4) The output, in any format, remains Prometheus specific anyway (though, again, noting that tools that understand prometheus format could not consume our novel JSON format).
It's an odd proposal that appears to satisfy no particular audience, I don't feel it's appropriate to add this to CouchDB in this form.
The goal seemed clear, "integrate with Prometheus", and the solution is equally clear, a new endpoint ("/_prometheus") that emits the documented Prometheus format with the 'text/plain; version="0.0.4' content type. The CouchDB project can accept or reject such a thing. Should Prometheus become old hat, we can remove the endpoint, losing no other functionality.
The code to interrogate CouchDB internals and produce the Prometheus output is useful regardless and could live on as a plugin or separate library (system administrators could add in the 'couchdb-prometheus-plugin' library as part of a custom 'make release') if the project were not to accept the work.
imo it is not a blocker that Prometheus does not speak JSON and that CouchDB does. This is an integration endpoint and, by definition, the two systems being integrated are not identical.
I would vote to accept a clean Prometheus endpoint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-961334939
Prometheus support shipped in 3.2.0
https://docs.couchdb.org/en/3.2.0/whatsnew/3.2.html#features-and-enhancements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] rnewson commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
rnewson commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-783534853
yes, that's right. and make the endpoint specifically about prometheus. the ad hoc attempts to make it generic aren't convincing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
nickva commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-961334939
Prometheus support shipped in 3.2.0
https://docs.couchdb.org/en/3.2.0/whatsnew/3.2.html#features-and-enhancements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva closed issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
nickva closed issue #3377:
URL: https://github.com/apache/couchdb/issues/3377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] tonysun83 commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
tonysun83 commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-785206669
@janl @rnewson @chewbranca updated the RFC to reflect Bob's comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva closed issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
nickva closed issue #3377:
URL: https://github.com/apache/couchdb/issues/3377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] janl commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
janl commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-783461468
+1 to all of @rnewson’s points
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] tonysun83 commented on issue #3377: RFC - Native Support for Prometheus Metrics Endpoint
Posted by GitBox <gi...@apache.org>.
tonysun83 commented on issue #3377:
URL: https://github.com/apache/couchdb/issues/3377#issuecomment-783512330
@rnewson @janl points taken. Let's continue the discussion here.
From these comments
- We're adding support for Prometheus but also inventing a new format for it that nothing understands.
- Negotiation by content-type when Prometheus doesn't have a registered one is a mistake (The accept= query parameter idea is better).
- The output, in any format, remains Prometheus specific anyway (though, again, noting that tools that understand prometheus format could not consume our novel JSON format).
It seems that both of you are not a big fan of having JSON as a return type and would rather only return `'text/plain; version="0.0.4' content type`. Am I understanding that correctly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org