You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2024/05/27 06:19:21 UTC
[I] PrometheusLayer: Add a new metrics called `bytes_duration_seconds` [opendal]
Xuanwo opened a new issue, #4644:
URL: https://github.com/apache/opendal/issues/4644
> Another blocker is that the layer observes the timer before returning `Writer`/`Reader`. So it doesn't record the time spent by the `Writer`/`Reader`.
The idea of tracking the duration for a reader and writer seems illogical to me because it might include irrelevant time periods. For instance, if you create a new writer and have them write every 10 seconds, measuring their duration doesn't make sense.
The current time only measures the cost of `read`/`write`, users can use this to know if they spent too much time on opening a `reader`/`writer`.
I think we can add a new metrics called `bytes_duration_seconds`, so we can observe the time we spent on reading/writing bytes.
_Originally posted by @Xuanwo in https://github.com/GreptimeTeam/greptimedb/issues/4041#issuecomment-2132673391_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PrometheusLayer: Add a new metrics called `bytes_duration_seconds` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4644:
URL: https://github.com/apache/opendal/issues/4644#issuecomment-2133621018
Yes, there are historical reasons why the two layers did not evolve simultaneously. And we do need to use a consistent way to record the request duration.
Our pattern will be:
- opendal_requests_duration_seconds{operation}: operation could be `Operation` or `ReadOperation`, like `read` and `Read::read`.
We don't need seperate metric for `open_writer` or `open_blocking_writer`, they should be covered by `opendal_requests_duration_seconds{operation="write"}` and `opendal_requests_duration_seconds{operation="blocking_write"}`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PrometheusLayer: Add a new metrics called `bytes_duration_seconds` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #4644: PrometheusLayer: Add a new metrics called `bytes_duration_seconds`
URL: https://github.com/apache/opendal/issues/4644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PrometheusLayer: Add a new metrics called `bytes_duration_seconds` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4644:
URL: https://github.com/apache/opendal/issues/4644#issuecomment-2149685788
Hi, @evenyag, your comment is correct. I have fixed in PR https://github.com/apache/opendal/pull/4691, welcome to check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PrometheusLayer: Add a new metrics called `bytes_duration_seconds` [opendal]
Posted by "evenyag (via GitHub)" <gi...@apache.org>.
evenyag commented on issue #4644:
URL: https://github.com/apache/opendal/issues/4644#issuecomment-2133571446
> The idea of tracking the duration for a reader and writer seems illogical to me because it might include irrelevant time periods. For instance, if you create a new writer and have them write every 10 seconds, measuring their duration doesn't make sense.
Understood. I also have the same concerns.
> The current time only measures the cost of read/write, users can use this to know if they spent too much time on opening a reader/writer.
>
> I think we can add a new metrics called bytes_duration_seconds, so we can observe the time we spent on reading/writing bytes.
I found that the three different metrics layer in OpenDAL observes the request duration in different ways.
## MetricsLayer
The `MetricsLayer` observes the time spent on reading/writing bytes to `opendal_requests_duration_seconds`. It doesn't observe the time creating the reader/writer.
https://github.com/apache/opendal/blob/bfe9e4499359edc5d5d9307efe26e78aafd50801/core/src/layers/metrics.rs#L787-L804
There is one exception: The block_write
https://github.com/apache/opendal/blob/bfe9e4499359edc5d5d9307efe26e78aafd50801/core/src/layers/metrics.rs#L636-L645
## PrometheusLayer
The `PrometheusLayer` only records the time creating the reader/writer to `requests_duration_seconds`
https://github.com/apache/opendal/blob/bfe9e4499359edc5d5d9307efe26e78aafd50801/core/src/layers/prometheus.rs#L345-L369
## PrometheusClientLayer
The `PrometheusClientLayer` records both the writer/reader creation time and bytes reading/writing time to `opendal_request_duration_seconds`
https://github.com/apache/opendal/blob/bfe9e4499359edc5d5d9307efe26e78aafd50801/core/src/layers/prometheus_client.rs#L577-L588
https://github.com/apache/opendal/blob/bfe9e4499359edc5d5d9307efe26e78aafd50801/core/src/layers/prometheus_client.rs#L637-L643
## Document
According to the document, users might expect to observe the time spent on reading/writing bytes from `requests_duration_seconds`.
https://github.com/apache/opendal/blob/bfe9e4499359edc5d5d9307efe26e78aafd50801/core/src/layers/metrics.rs#L37-L48
https://github.com/apache/opendal/blob/bfe9e4499359edc5d5d9307efe26e78aafd50801/core/src/layers/prometheus.rs#L49
Maybe we should add a new metric or label to record the cost of opening a reader/writer? e.g. label value `open_writer`/`open_blocking_writer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PrometheusLayer: Add a new metrics called `bytes_duration_seconds` [opendal]
Posted by "evenyag (via GitHub)" <gi...@apache.org>.
evenyag commented on issue #4644:
URL: https://github.com/apache/opendal/issues/4644#issuecomment-2137463741
Do we still need `bytes_duration_seconds`?
> opendal_requests_duration_seconds{operation}: operation could be Operation or ReadOperation, like read and Read::read.
Do you mean that `opendal_requests_duration_seconds{operation="read"}` measures the cost of opening a reader, and `opendal_requests_duration_seconds{operation="Read::read"}` measures the time of reading bytes?
Will we have a metric `opendal_requests_duration_seconds{operation="Write::write"}` to record the time of writing bytes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org