You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Siyu Lin <si...@unity3d.com> on 2022/04/04 23:44:26 UTC
Histogram metrics in Dataflow/Beam
Hi Beam community,
I am wondering if there is histogram metrics available (or alternative recommendations) for showing up quantiles. We have counter metrics already but we would also like to see some quantiles for different values.
Thanks a lot!
Siyu
Re: Histogram metrics in Dataflow/Beam
Posted by Reuven Lax <re...@google.com>.
Also, if this is a streaming pipeline, note that the Distribution metric
does not track a histogram. i.e. MAX is the largest value ever seen, MIN is
the smallest value ever seen, and MEAN is the mean value over all time.
On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <si...@unity3d.com> wrote:
> Hi Jeff,
>
> Thanks so much for your quick responses. It is unfortunate that histogram
> is unavailable in dataflow. Do you know if there are any workaround? Or do
> you think it is plausible if we can use runner v2 and customize the image
> with Prometheus exporter?
>
> Thanks again!
> Siyu
>
>
> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>
>
> Siyu - The Beam metrics interface includes the Distribution metric type
> which can be used for histograms:
>
> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>
> Particulars of support depend on the runner. For Cloud Dataflow, the
> reported values are MAX, MIN, MEAN, and COUNT, so no support for
> finer-grained percentiles:
>
>
> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>
> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>
>> Hi Beam community,
>>
>> I am wondering if there is histogram metrics available (or alternative
>> recommendations) for showing up quantiles. We have counter metrics already
>> but we would also like to see some quantiles for different values.
>>
>> Thanks a lot!
>> Siyu
>
>
Re: Histogram metrics in Dataflow/Beam
Posted by Siyu Lin <si...@unity3d.com>.
Got it! Thanks so much, and will try it out!
-siyu
>
> On May 6, 2022, at 4:53 PM, Pablo Estrada <pa...@google.com> wrote:
>
>
> Hi Siyu!
> Yeah, that's a good question!
>
> Depending on the properties of the client that you're using, you can create it as a class/static variable rather than an instance variable:
>
> class MyPrometheusDoFn extends DoFn<...> {
> static PrometheusClient client = Prometheus.client(); // Or whatever
> }
>
> You just need to consider that in this scenario, the variable `client` may be used by many threads at the same time - so you need to make sure that PrometheusClient is thread safe, or you need to synchronize accesses to it. (maybe it supports some kind of async call?)
>
> Let me know if that makes sense?
> Best
> -P.
>
>> On Fri, May 6, 2022 at 4:33 PM Siyu Lin <si...@unity3d.com> wrote:
>> sorry for typo “recommissioned way” should be “recommended way”
>>
>>
>>>> On May 6, 2022, at 4:31 PM, Siyu Lin <si...@unity3d.com> wrote:
>>>>
>>>
>>> Hi Pablo,
>>>
>>> Thanks so much for your explanation!
>>>
>>> Also, for prom client in the do fn, do we need to initialize them in the setup code? My concern is that if we have this initialization in each do fn, we might overwhelm the memory. I think we can pass them as the parameter? Do you have a recommissioned way to do that not to hurt the performance?
>>>
>>> Will try that out to see if this is workable!
>>>
>>> Siyu
>>>
>>>>
>>>>> On May 6, 2022, at 3:35 PM, Pablo Estrada <pa...@google.com> wrote:
>>>>>
>>>>
>>>> ah - no, I am not aware of anyone running side containers.
>>>>
>>>> You can add libraries to your container that will be available, so that you can import prometheus client libraries in your DoFns and export metrics directly 'by hand' (e.g. processElement(..) { prometheusCliient.reportMetric("name", TYPE, value); })
>>>>
>>>> If you need to run another program in your worker - I suppose you can add new binaries that run next to your Beam worker processes (in the same container, and you have to start them up in the entrypoint of your Dockerfile). I've never seen anyone do this, but it sounds quite doable.
>>>>
>>>> Do any of these options work for you?
>>>> Best
>>>> -P.
>>>>
>>>>> On Fri, May 6, 2022 at 3:16 PM Siyu Lin <si...@unity3d.com> wrote:
>>>>> Hi Pablo,
>>>>>
>>>>> No worries at all!
>>>>>
>>>>> Was wondering if “add any dependencies” means to add a side container to the dataflow runner? Like prometheus scraper?
>>>>>
>>>>> If so, is it only working for dataflow runner v2? We have not upgraded to v2 yet so it might be hard to do that in v1.
>>>>>
>>>>> Thanks so much!
>>>>>
>>>>> Siyu
>>>>>
>>>>>> On May 6, 2022, at 11:57 AM, Pablo Estrada <pa...@google.com> wrote:
>>>>>>
>>>>>> Sorry about the delay!
>>>>>>
>>>>>> Yes, you can add any dependencies to your image - and you can add custom reporting of metrics that you're tracking directly. That may help?
>>>>>>
>>>>>>> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <si...@unity3d.com> wrote:
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> Thanks so much for your quick responses. It is unfortunate that histogram is unavailable in dataflow. Do you know if there are any workaround? Or do you think it is plausible if we can use runner v2 and customize the image with Prometheus exporter?
>>>>>>>
>>>>>>> Thanks again!
>>>>>>> Siyu
>>>>>>>
>>>>>>>>
>>>>>>>>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>> Siyu - The Beam metrics interface includes the Distribution metric type which can be used for histograms:
>>>>>>>>
>>>>>>>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>>>>>>>
>>>>>>>> Particulars of support depend on the runner. For Cloud Dataflow, the reported values are MAX, MIN, MEAN, and COUNT, so no support for finer-grained percentiles:
>>>>>>>>
>>>>>>>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>>>>>>>
>>>>>>>>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>>>>>>>>> Hi Beam community,
>>>>>>>>>
>>>>>>>>> I am wondering if there is histogram metrics available (or alternative recommendations) for showing up quantiles. We have counter metrics already but we would also like to see some quantiles for different values.
>>>>>>>>>
>>>>>>>>> Thanks a lot!
>>>>>>>>> Siyu
>>>>>
Re: Histogram metrics in Dataflow/Beam
Posted by Pablo Estrada <pa...@google.com>.
Hi Siyu!
Yeah, that's a good question!
Depending on the properties of the client that you're using, you can create
it as a class/static variable rather than an instance variable:
class MyPrometheusDoFn extends DoFn<...> {
static PrometheusClient client = Prometheus.client(); // Or whatever
}
You just need to consider that in this scenario, the variable `client` may
be used by many threads at the same time - so you need to make sure that
PrometheusClient is thread safe, or you need to synchronize accesses to it.
(maybe it supports some kind of async call?)
Let me know if that makes sense?
Best
-P.
On Fri, May 6, 2022 at 4:33 PM Siyu Lin <si...@unity3d.com> wrote:
> sorry for typo “recommissioned way” should be “recommended way”
>
>
> On May 6, 2022, at 4:31 PM, Siyu Lin <si...@unity3d.com> wrote:
>
>
> Hi Pablo,
>
> Thanks so much for your explanation!
>
> Also, for prom client in the do fn, do we need to initialize them in the
> setup code? My concern is that if we have this initialization in each do
> fn, we might overwhelm the memory. I think we can pass them as the
> parameter? Do you have a recommissioned way to do that not to hurt the
> performance?
>
> Will try that out to see if this is workable!
>
> Siyu
>
>
> On May 6, 2022, at 3:35 PM, Pablo Estrada <pa...@google.com> wrote:
>
>
> ah - no, I am not aware of anyone running side containers.
>
> You can add libraries to your container that will be available, so that
> you can import prometheus client libraries in your DoFns and export metrics
> directly 'by hand' (e.g. processElement(..) {
> prometheusCliient.reportMetric("name", TYPE, value); })
>
> If you need to run another program in your worker - I suppose you can add
> new binaries that run next to your Beam worker processes (in the same
> container, and you have to start them up in the entrypoint of your
> Dockerfile). I've never seen anyone do this, but it sounds quite doable.
>
> Do any of these options work for you?
> Best
> -P.
>
> On Fri, May 6, 2022 at 3:16 PM Siyu Lin <si...@unity3d.com> wrote:
>
>> Hi Pablo,
>>
>> No worries at all!
>>
>> Was wondering if “add any dependencies” means to add a side container to
>> the dataflow runner? Like prometheus scraper?
>>
>> If so, is it only working for dataflow runner v2? We have not upgraded to
>> v2 yet so it might be hard to do that in v1.
>>
>> Thanks so much!
>>
>> Siyu
>>
>> On May 6, 2022, at 11:57 AM, Pablo Estrada <pa...@google.com> wrote:
>>
>> Sorry about the delay!
>>
>> Yes, you can add any dependencies to your image - and you can add custom
>> reporting of metrics that you're tracking directly. That may help?
>>
>> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <si...@unity3d.com> wrote:
>>
>>> Hi Jeff,
>>>
>>> Thanks so much for your quick responses. It is unfortunate that
>>> histogram is unavailable in dataflow. Do you know if there are any
>>> workaround? Or do you think it is plausible if we can use runner v2 and
>>> customize the image with Prometheus exporter?
>>>
>>> Thanks again!
>>> Siyu
>>>
>>>
>>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>>>
>>>
>>> Siyu - The Beam metrics interface includes the Distribution metric type
>>> which can be used for histograms:
>>>
>>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>>
>>> Particulars of support depend on the runner. For Cloud Dataflow, the
>>> reported values are MAX, MIN, MEAN, and COUNT, so no support for
>>> finer-grained percentiles:
>>>
>>>
>>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>>
>>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>>>
>>>> Hi Beam community,
>>>>
>>>> I am wondering if there is histogram metrics available (or alternative
>>>> recommendations) for showing up quantiles. We have counter metrics already
>>>> but we would also like to see some quantiles for different values.
>>>>
>>>> Thanks a lot!
>>>> Siyu
>>>
>>>
>>
Re: Histogram metrics in Dataflow/Beam
Posted by Siyu Lin <si...@unity3d.com>.
sorry for typo “recommissioned way” should be “recommended way”
> On May 6, 2022, at 4:31 PM, Siyu Lin <si...@unity3d.com> wrote:
>
>
> Hi Pablo,
>
> Thanks so much for your explanation!
>
> Also, for prom client in the do fn, do we need to initialize them in the setup code? My concern is that if we have this initialization in each do fn, we might overwhelm the memory. I think we can pass them as the parameter? Do you have a recommissioned way to do that not to hurt the performance?
>
> Will try that out to see if this is workable!
>
> Siyu
>
>>
>>> On May 6, 2022, at 3:35 PM, Pablo Estrada <pa...@google.com> wrote:
>>>
>>
>> ah - no, I am not aware of anyone running side containers.
>>
>> You can add libraries to your container that will be available, so that you can import prometheus client libraries in your DoFns and export metrics directly 'by hand' (e.g. processElement(..) { prometheusCliient.reportMetric("name", TYPE, value); })
>>
>> If you need to run another program in your worker - I suppose you can add new binaries that run next to your Beam worker processes (in the same container, and you have to start them up in the entrypoint of your Dockerfile). I've never seen anyone do this, but it sounds quite doable.
>>
>> Do any of these options work for you?
>> Best
>> -P.
>>
>>> On Fri, May 6, 2022 at 3:16 PM Siyu Lin <si...@unity3d.com> wrote:
>>> Hi Pablo,
>>>
>>> No worries at all!
>>>
>>> Was wondering if “add any dependencies” means to add a side container to the dataflow runner? Like prometheus scraper?
>>>
>>> If so, is it only working for dataflow runner v2? We have not upgraded to v2 yet so it might be hard to do that in v1.
>>>
>>> Thanks so much!
>>>
>>> Siyu
>>>
>>>> On May 6, 2022, at 11:57 AM, Pablo Estrada <pa...@google.com> wrote:
>>>>
>>>> Sorry about the delay!
>>>>
>>>> Yes, you can add any dependencies to your image - and you can add custom reporting of metrics that you're tracking directly. That may help?
>>>>
>>>>> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <si...@unity3d.com> wrote:
>>>>> Hi Jeff,
>>>>>
>>>>> Thanks so much for your quick responses. It is unfortunate that histogram is unavailable in dataflow. Do you know if there are any workaround? Or do you think it is plausible if we can use runner v2 and customize the image with Prometheus exporter?
>>>>>
>>>>> Thanks again!
>>>>> Siyu
>>>>>
>>>>>>
>>>>>>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>>>>>>>
>>>>>>
>>>>>> Siyu - The Beam metrics interface includes the Distribution metric type which can be used for histograms:
>>>>>>
>>>>>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>>>>>
>>>>>> Particulars of support depend on the runner. For Cloud Dataflow, the reported values are MAX, MIN, MEAN, and COUNT, so no support for finer-grained percentiles:
>>>>>>
>>>>>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>>>>>
>>>>>>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>>>>>>> Hi Beam community,
>>>>>>>
>>>>>>> I am wondering if there is histogram metrics available (or alternative recommendations) for showing up quantiles. We have counter metrics already but we would also like to see some quantiles for different values.
>>>>>>>
>>>>>>> Thanks a lot!
>>>>>>> Siyu
>>>
Re: Histogram metrics in Dataflow/Beam
Posted by Siyu Lin <si...@unity3d.com>.
Hi Pablo,
Thanks so much for your explanation!
Also, for prom client in the do fn, do we need to initialize them in the setup code? My concern is that if we have this initialization in each do fn, we might overwhelm the memory. I think we can pass them as the parameter? Do you have a recommissioned way to do that not to hurt the performance?
Will try that out to see if this is workable!
Siyu
>
> On May 6, 2022, at 3:35 PM, Pablo Estrada <pa...@google.com> wrote:
>
>
> ah - no, I am not aware of anyone running side containers.
>
> You can add libraries to your container that will be available, so that you can import prometheus client libraries in your DoFns and export metrics directly 'by hand' (e.g. processElement(..) { prometheusCliient.reportMetric("name", TYPE, value); })
>
> If you need to run another program in your worker - I suppose you can add new binaries that run next to your Beam worker processes (in the same container, and you have to start them up in the entrypoint of your Dockerfile). I've never seen anyone do this, but it sounds quite doable.
>
> Do any of these options work for you?
> Best
> -P.
>
>> On Fri, May 6, 2022 at 3:16 PM Siyu Lin <si...@unity3d.com> wrote:
>> Hi Pablo,
>>
>> No worries at all!
>>
>> Was wondering if “add any dependencies” means to add a side container to the dataflow runner? Like prometheus scraper?
>>
>> If so, is it only working for dataflow runner v2? We have not upgraded to v2 yet so it might be hard to do that in v1.
>>
>> Thanks so much!
>>
>> Siyu
>>
>>> On May 6, 2022, at 11:57 AM, Pablo Estrada <pa...@google.com> wrote:
>>>
>>> Sorry about the delay!
>>>
>>> Yes, you can add any dependencies to your image - and you can add custom reporting of metrics that you're tracking directly. That may help?
>>>
>>>> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <si...@unity3d.com> wrote:
>>>> Hi Jeff,
>>>>
>>>> Thanks so much for your quick responses. It is unfortunate that histogram is unavailable in dataflow. Do you know if there are any workaround? Or do you think it is plausible if we can use runner v2 and customize the image with Prometheus exporter?
>>>>
>>>> Thanks again!
>>>> Siyu
>>>>
>>>>>
>>>>>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>>>>>>
>>>>>
>>>>> Siyu - The Beam metrics interface includes the Distribution metric type which can be used for histograms:
>>>>>
>>>>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>>>>
>>>>> Particulars of support depend on the runner. For Cloud Dataflow, the reported values are MAX, MIN, MEAN, and COUNT, so no support for finer-grained percentiles:
>>>>>
>>>>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>>>>
>>>>>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>>>>>> Hi Beam community,
>>>>>>
>>>>>> I am wondering if there is histogram metrics available (or alternative recommendations) for showing up quantiles. We have counter metrics already but we would also like to see some quantiles for different values.
>>>>>>
>>>>>> Thanks a lot!
>>>>>> Siyu
>>
Re: Histogram metrics in Dataflow/Beam
Posted by Pablo Estrada <pa...@google.com>.
ah - no, I am not aware of anyone running side containers.
You can add libraries to your container that will be available, so that you
can import prometheus client libraries in your DoFns and export metrics
directly 'by hand' (e.g. processElement(..) {
prometheusCliient.reportMetric("name", TYPE, value); })
If you need to run another program in your worker - I suppose you can add
new binaries that run next to your Beam worker processes (in the same
container, and you have to start them up in the entrypoint of your
Dockerfile). I've never seen anyone do this, but it sounds quite doable.
Do any of these options work for you?
Best
-P.
On Fri, May 6, 2022 at 3:16 PM Siyu Lin <si...@unity3d.com> wrote:
> Hi Pablo,
>
> No worries at all!
>
> Was wondering if “add any dependencies” means to add a side container to
> the dataflow runner? Like prometheus scraper?
>
> If so, is it only working for dataflow runner v2? We have not upgraded to
> v2 yet so it might be hard to do that in v1.
>
> Thanks so much!
>
> Siyu
>
> On May 6, 2022, at 11:57 AM, Pablo Estrada <pa...@google.com> wrote:
>
> Sorry about the delay!
>
> Yes, you can add any dependencies to your image - and you can add custom
> reporting of metrics that you're tracking directly. That may help?
>
> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <si...@unity3d.com> wrote:
>
>> Hi Jeff,
>>
>> Thanks so much for your quick responses. It is unfortunate that histogram
>> is unavailable in dataflow. Do you know if there are any workaround? Or do
>> you think it is plausible if we can use runner v2 and customize the image
>> with Prometheus exporter?
>>
>> Thanks again!
>> Siyu
>>
>>
>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>>
>>
>> Siyu - The Beam metrics interface includes the Distribution metric type
>> which can be used for histograms:
>>
>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>
>> Particulars of support depend on the runner. For Cloud Dataflow, the
>> reported values are MAX, MIN, MEAN, and COUNT, so no support for
>> finer-grained percentiles:
>>
>>
>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>
>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>>
>>> Hi Beam community,
>>>
>>> I am wondering if there is histogram metrics available (or alternative
>>> recommendations) for showing up quantiles. We have counter metrics already
>>> but we would also like to see some quantiles for different values.
>>>
>>> Thanks a lot!
>>> Siyu
>>
>>
>
Re: Histogram metrics in Dataflow/Beam
Posted by Siyu Lin <si...@unity3d.com>.
Hi Pablo,
No worries at all!
Was wondering if “add any dependencies” means to add a side container to the dataflow runner? Like prometheus scraper?
If so, is it only working for dataflow runner v2? We have not upgraded to v2 yet so it might be hard to do that in v1.
Thanks so much!
Siyu
> On May 6, 2022, at 11:57 AM, Pablo Estrada <pa...@google.com> wrote:
>
> Sorry about the delay!
>
> Yes, you can add any dependencies to your image - and you can add custom reporting of metrics that you're tracking directly. That may help?
>
> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <siyulin@unity3d.com <ma...@unity3d.com>> wrote:
> Hi Jeff,
>
> Thanks so much for your quick responses. It is unfortunate that histogram is unavailable in dataflow. Do you know if there are any workaround? Or do you think it is plausible if we can use runner v2 and customize the image with Prometheus exporter?
>
> Thanks again!
> Siyu
>
>>
>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jklukas@mozilla.com <ma...@mozilla.com>> wrote:
>>
>>
>> Siyu - The Beam metrics interface includes the Distribution metric type which can be used for histograms:
>>
>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics <https://beam.apache.org/documentation/programming-guide/#types-of-metrics>
>>
>> Particulars of support depend on the runner. For Cloud Dataflow, the reported values are MAX, MIN, MEAN, and COUNT, so no support for finer-grained percentiles:
>>
>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics <https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics>
>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <siyulin@unity3d.com <ma...@unity3d.com>> wrote:
>> Hi Beam community,
>>
>> I am wondering if there is histogram metrics available (or alternative recommendations) for showing up quantiles. We have counter metrics already but we would also like to see some quantiles for different values.
>>
>> Thanks a lot!
>> Siyu
Re: Histogram metrics in Dataflow/Beam
Posted by Pablo Estrada <pa...@google.com>.
Sorry about the delay!
Yes, you can add any dependencies to your image - and you can add custom
reporting of metrics that you're tracking directly. That may help?
On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <si...@unity3d.com> wrote:
> Hi Jeff,
>
> Thanks so much for your quick responses. It is unfortunate that histogram
> is unavailable in dataflow. Do you know if there are any workaround? Or do
> you think it is plausible if we can use runner v2 and customize the image
> with Prometheus exporter?
>
> Thanks again!
> Siyu
>
>
> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>
>
> Siyu - The Beam metrics interface includes the Distribution metric type
> which can be used for histograms:
>
> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>
> Particulars of support depend on the runner. For Cloud Dataflow, the
> reported values are MAX, MIN, MEAN, and COUNT, so no support for
> finer-grained percentiles:
>
>
> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>
> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>
>> Hi Beam community,
>>
>> I am wondering if there is histogram metrics available (or alternative
>> recommendations) for showing up quantiles. We have counter metrics already
>> but we would also like to see some quantiles for different values.
>>
>> Thanks a lot!
>> Siyu
>
>
Re: Histogram metrics in Dataflow/Beam
Posted by Siyu Lin <si...@unity3d.com>.
Hi Jeff,
Thanks so much for your quick responses. It is unfortunate that histogram is unavailable in dataflow. Do you know if there are any workaround? Or do you think it is plausible if we can use runner v2 and customize the image with Prometheus exporter?
Thanks again!
Siyu
>
> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jk...@mozilla.com> wrote:
>
>
> Siyu - The Beam metrics interface includes the Distribution metric type which can be used for histograms:
>
> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>
> Particulars of support depend on the runner. For Cloud Dataflow, the reported values are MAX, MIN, MEAN, and COUNT, so no support for finer-grained percentiles:
>
> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>
>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
>> Hi Beam community,
>>
>> I am wondering if there is histogram metrics available (or alternative recommendations) for showing up quantiles. We have counter metrics already but we would also like to see some quantiles for different values.
>>
>> Thanks a lot!
>> Siyu
Re: Histogram metrics in Dataflow/Beam
Posted by Jeff Klukas <jk...@mozilla.com>.
Siyu - The Beam metrics interface includes the Distribution metric type
which can be used for histograms:
https://beam.apache.org/documentation/programming-guide/#types-of-metrics
Particulars of support depend on the runner. For Cloud Dataflow, the
reported values are MAX, MIN, MEAN, and COUNT, so no support for
finer-grained percentiles:
https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <si...@unity3d.com> wrote:
> Hi Beam community,
>
> I am wondering if there is histogram metrics available (or alternative
> recommendations) for showing up quantiles. We have counter metrics already
> but we would also like to see some quantiles for different values.
>
> Thanks a lot!
> Siyu