You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Etienne Chauchot <ec...@apache.org> on 2020/03/02 10:47:29 UTC
Re: Beam Emitted Metrics Reference
Hi,
There is a doc about metrics here:
https://beam.apache.org/documentation/programming-guide/#metrics
You can also export the metrics to sinks (REST http endpoint and
Graphite), see MetricsOptions class for configuration.
Still, there is no doc for export on website, I'll add some
Best
Etienne
On 28/02/2020 01:07, Pablo Estrada wrote:
> Hi Daniel!
> I think +Alex Amato <ma...@google.com> had tried to have an
> inventory of metrics at some point.
> Other than that, I don't think we have a document outlining them.
>
> Can you talk about what you plan to do with them? Do you plan to
> export them somehow? Do you plan to add your own?
> Best
> -P.
>
> On Thu, Feb 27, 2020 at 11:33 AM Daniel Chen <danxich@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi all,
>
> I some questions about the reference to the framework metrics
> emitted by Beam. I would like to leverage these metrics to allow
> better monitoring of by Beam jobs but cannot find any references
> to the description or a complete set of emitted metrics.
>
> Do we have this information documented anywhere?
>
> Thanks,
> Daniel
>
Re: Beam Emitted Metrics Reference
Posted by Kenneth Knowles <ke...@apache.org>.
Also seems like each IO would benefit from an entry in a Transform Catalog
with description of any IO-specific metrics it emits. (technically these
may not be what you mean by "framework-emitted metrics")
Kenn
On Mon, Mar 2, 2020 at 9:40 AM Alex Amato <aj...@google.com> wrote:
> MonitoringInfoSpecs is effectively a list of metrics
> <https://github.com/apache/beam/blob/c0b60195b17b7cdc46d6ad6548cd41a967e71cde/model/pipeline/src/main/proto/metrics.proto#L60>,
> but its purpose is to simply define how SDKs should populate MonitoringInfo
> protos for a RunnerHarness to interpret.
>
> These metrics are provided by the Java and Python SDKs, and Go will soon
> provide all of them as well.
>
> But there is no requirement for a particular runner to support any of
> these metrics. The DataflowRunner will support these and they metrics are
> accessible via Dataflow APIs. I am not sure the state of other runners.
>
>
>
>
> On Mon, Mar 2, 2020 at 2:47 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
>> Hi,
>>
>> There is a doc about metrics here:
>> https://beam.apache.org/documentation/programming-guide/#metrics
>>
>> You can also export the metrics to sinks (REST http endpoint and
>> Graphite), see MetricsOptions class for configuration.
>>
>> Still, there is no doc for export on website, I'll add some
>>
>> Best
>>
>> Etienne
>> On 28/02/2020 01:07, Pablo Estrada wrote:
>>
>> Hi Daniel!
>> I think +Alex Amato <aj...@google.com> had tried to have an inventory
>> of metrics at some point.
>> Other than that, I don't think we have a document outlining them.
>>
>> Can you talk about what you plan to do with them? Do you plan to export
>> them somehow? Do you plan to add your own?
>> Best
>> -P.
>>
>> On Thu, Feb 27, 2020 at 11:33 AM Daniel Chen <da...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I some questions about the reference to the framework metrics emitted by
>>> Beam. I would like to leverage these metrics to allow better monitoring of
>>> by Beam jobs but cannot find any references to the description or a
>>> complete set of emitted metrics.
>>>
>>> Do we have this information documented anywhere?
>>>
>>> Thanks,
>>> Daniel
>>>
>>
Re: Beam Emitted Metrics Reference
Posted by Alex Amato <aj...@google.com>.
MonitoringInfoSpecs is effectively a list of metrics
<https://github.com/apache/beam/blob/c0b60195b17b7cdc46d6ad6548cd41a967e71cde/model/pipeline/src/main/proto/metrics.proto#L60>,
but its purpose is to simply define how SDKs should populate MonitoringInfo
protos for a RunnerHarness to interpret.
These metrics are provided by the Java and Python SDKs, and Go will soon
provide all of them as well.
But there is no requirement for a particular runner to support any of these
metrics. The DataflowRunner will support these and they metrics are
accessible via Dataflow APIs. I am not sure the state of other runners.
On Mon, Mar 2, 2020 at 2:47 AM Etienne Chauchot <ec...@apache.org>
wrote:
> Hi,
>
> There is a doc about metrics here:
> https://beam.apache.org/documentation/programming-guide/#metrics
>
> You can also export the metrics to sinks (REST http endpoint and
> Graphite), see MetricsOptions class for configuration.
>
> Still, there is no doc for export on website, I'll add some
>
> Best
>
> Etienne
> On 28/02/2020 01:07, Pablo Estrada wrote:
>
> Hi Daniel!
> I think +Alex Amato <aj...@google.com> had tried to have an inventory
> of metrics at some point.
> Other than that, I don't think we have a document outlining them.
>
> Can you talk about what you plan to do with them? Do you plan to export
> them somehow? Do you plan to add your own?
> Best
> -P.
>
> On Thu, Feb 27, 2020 at 11:33 AM Daniel Chen <da...@gmail.com> wrote:
>
>> Hi all,
>>
>> I some questions about the reference to the framework metrics emitted by
>> Beam. I would like to leverage these metrics to allow better monitoring of
>> by Beam jobs but cannot find any references to the description or a
>> complete set of emitted metrics.
>>
>> Do we have this information documented anywhere?
>>
>> Thanks,
>> Daniel
>>
>