You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2019/01/31 19:45:55 UTC
[Proposal] Get Metrics API: Metric Extraction via proto RPC API.
Hello Beam,
Robert Ryan and I have been designing a metric extraction API for Beam.
Please take a look at this design, I would love to get more feedback on
this to improve the design.
https://s.apache.org/get-metrics-api
The primary goal of this proposal is to offer a simple way to obtain all
the metrics for a job. The following issues are addressed:
-
The current design requires implementing metric querying for every
runner+language combination.
-
Duplication of MetricResult related classes in each language.
-
The existing MetricResult format only allows querying metrics defined by
a namespace, name and step, and does not allow generalized labelling as
used by MonitoringInfos.
-
Enhance Beam’s ability to integration test new metrics
Thank for taking a look,
Alex
Re: [Proposal] Get Metrics API: Metric Extraction via proto RPC API.
Posted by Ryan Williams <ry...@runsascoded.com>.
Following up here:
A few PRs have gone in related to this proposal (see BEAM-4775
<https://issues.apache.org/jira/browse/BEAM-4775>), and Robert, Alex,
Mikhail, and I are discussing some questions that have come up along the
way.
I wrote up the state of things in this doc
<https://docs.google.com/document/d/1m83TsFvJbOlcLfXVXprQm1B7vUakhbLZMzuRrOHWnTg/edit#heading=h.faqan9rjc6dm>.
It describes PRs that are in-flight as well as some further downstream
questions:
- Should metrics' {namespace,name} be stored in the URN, or as
MonitoringInfo "labels"?
- Should "attempted"/"committed" be surfaced as proto fields, or also
made into a MonitoringInfo label?
- Also, is there a discrepancy in what those terms mean, between
older "physical"/"logical" semantics and new FN-API "bundle progress" /
"bundle result" ones?
It's not really a design doc at this stage, but I wanted to post it in case
others are interested to follow or discuss. If we get agreement on these
questions, we can update the previous design doc of record covering this
work (https://s.apache.org/get-metrics-api).
Thanks,
-Ryan
On Mon, Feb 4, 2019 at 2:26 PM Robert Bradshaw <ro...@google.com> wrote:
> To summarize for the list, the plan of record is:
>
> The MonitoringInfo proto will be used again in this querying API, so the
> metric format SDKs report will also be used when extracting metrics for a
> job.
>
> // Job Service for running RunnerAPI pipelines
> service JobService {
> ...
> rpc GetJobMetrics (GetJobMetricsRequest) returns (GetJobMetricsResponse);
> }
>
> message GetJobMetricsRequest {
> string job_id = 1; // (required)
> }
>
> message GetJobMetricsResponse {
> // (Optional) The aggregated value of the metric based on the in
> flight work.
> // SDKs optionally report these metrics in the
> ProcessBundleProgressResponse.
> MonitoringInfo attempted_metric_result = 1;
> // (Required) The aggregated value of the metric based on the
> completed work.
> // SDKs report these metrics in the ProcessBundleResponse.
> MonitoringInfo committed_metric_result = 2;
> }
> The new rpc in beam_job_metrics.proto.
>
> SDKs will continue to implement filtering of Metrics, by providing their
> own language specific convenience functions to filter and obtain metrics.
> In Java for example, the MetricResult and MetricFilter format interfaces
> will continue to exist as interface to filter metrics for a specific step,
> user namespace, username, etc.
>
>
> Looking forward to seeing this happen.
>
>
> On Mon, Feb 4, 2019 at 8:17 PM Alex Amato <aj...@google.com> wrote:
>
>> Done,it's on the website now.
>>
>> Ryan and I will move forward with the plan in this plan. If there are any
>> major objections to this plan, please let us know by weds. Suggestions will
>> be welcome later as well, as we are happy to iterate on this. But we will
>> be proceeding with some of Ryan's PRs based on this design.
>>
>> On Thu, Jan 31, 2019 at 12:54 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Please don't forget to add this document to the design documents webpage.
>>>
>>> On Thu, Jan 31, 2019 at 8:46 PM Alex Amato <aj...@google.com> wrote:
>>> >
>>> > Hello Beam,
>>> >
>>> > Robert Ryan and I have been designing a metric extraction API for
>>> Beam. Please take a look at this design, I would love to get more feedback
>>> on this to improve the design.
>>> >
>>> > https://s.apache.org/get-metrics-api
>>> >
>>> > The primary goal of this proposal is to offer a simple way to obtain
>>> all the metrics for a job. The following issues are addressed:
>>> >
>>> > The current design requires implementing metric querying for every
>>> runner+language combination.
>>> >
>>> > Duplication of MetricResult related classes in each language.
>>> >
>>> > The existing MetricResult format only allows querying metrics defined
>>> by a namespace, name and step, and does not allow generalized labelling as
>>> used by MonitoringInfos.
>>> >
>>> > Enhance Beam’s ability to integration test new metrics
>>> >
>>> >
>>> > Thank for taking a look,
>>> > Alex
>>>
>>
Re: [Proposal] Get Metrics API: Metric Extraction via proto RPC API.
Posted by Robert Bradshaw <ro...@google.com>.
To summarize for the list, the plan of record is:
The MonitoringInfo proto will be used again in this querying API, so the
metric format SDKs report will also be used when extracting metrics for a
job.
// Job Service for running RunnerAPI pipelines
service JobService {
...
rpc GetJobMetrics (GetJobMetricsRequest) returns (GetJobMetricsResponse);
}
message GetJobMetricsRequest {
string job_id = 1; // (required)
}
message GetJobMetricsResponse {
// (Optional) The aggregated value of the metric based on the in
flight work.
// SDKs optionally report these metrics in the
ProcessBundleProgressResponse.
MonitoringInfo attempted_metric_result = 1;
// (Required) The aggregated value of the metric based on the
completed work.
// SDKs report these metrics in the ProcessBundleResponse.
MonitoringInfo committed_metric_result = 2;
}
The new rpc in beam_job_metrics.proto.
SDKs will continue to implement filtering of Metrics, by providing their
own language specific convenience functions to filter and obtain metrics.
In Java for example, the MetricResult and MetricFilter format interfaces
will continue to exist as interface to filter metrics for a specific step,
user namespace, username, etc.
Looking forward to seeing this happen.
On Mon, Feb 4, 2019 at 8:17 PM Alex Amato <aj...@google.com> wrote:
> Done,it's on the website now.
>
> Ryan and I will move forward with the plan in this plan. If there are any
> major objections to this plan, please let us know by weds. Suggestions will
> be welcome later as well, as we are happy to iterate on this. But we will
> be proceeding with some of Ryan's PRs based on this design.
>
> On Thu, Jan 31, 2019 at 12:54 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Please don't forget to add this document to the design documents webpage.
>>
>> On Thu, Jan 31, 2019 at 8:46 PM Alex Amato <aj...@google.com> wrote:
>> >
>> > Hello Beam,
>> >
>> > Robert Ryan and I have been designing a metric extraction API for Beam.
>> Please take a look at this design, I would love to get more feedback on
>> this to improve the design.
>> >
>> > https://s.apache.org/get-metrics-api
>> >
>> > The primary goal of this proposal is to offer a simple way to obtain
>> all the metrics for a job. The following issues are addressed:
>> >
>> > The current design requires implementing metric querying for every
>> runner+language combination.
>> >
>> > Duplication of MetricResult related classes in each language.
>> >
>> > The existing MetricResult format only allows querying metrics defined
>> by a namespace, name and step, and does not allow generalized labelling as
>> used by MonitoringInfos.
>> >
>> > Enhance Beam’s ability to integration test new metrics
>> >
>> >
>> > Thank for taking a look,
>> > Alex
>>
>
Re: [Proposal] Get Metrics API: Metric Extraction via proto RPC API.
Posted by Alex Amato <aj...@google.com>.
Done,it's on the website now.
Ryan and I will move forward with the plan in this plan. If there are any
major objections to this plan, please let us know by weds. Suggestions will
be welcome later as well, as we are happy to iterate on this. But we will
be proceeding with some of Ryan's PRs based on this design.
On Thu, Jan 31, 2019 at 12:54 PM Ismaël Mejía <ie...@gmail.com> wrote:
> Please don't forget to add this document to the design documents webpage.
>
> On Thu, Jan 31, 2019 at 8:46 PM Alex Amato <aj...@google.com> wrote:
> >
> > Hello Beam,
> >
> > Robert Ryan and I have been designing a metric extraction API for Beam.
> Please take a look at this design, I would love to get more feedback on
> this to improve the design.
> >
> > https://s.apache.org/get-metrics-api
> >
> > The primary goal of this proposal is to offer a simple way to obtain all
> the metrics for a job. The following issues are addressed:
> >
> > The current design requires implementing metric querying for every
> runner+language combination.
> >
> > Duplication of MetricResult related classes in each language.
> >
> > The existing MetricResult format only allows querying metrics defined by
> a namespace, name and step, and does not allow generalized labelling as
> used by MonitoringInfos.
> >
> > Enhance Beam’s ability to integration test new metrics
> >
> >
> > Thank for taking a look,
> > Alex
>
Re: [Proposal] Get Metrics API: Metric Extraction via proto RPC API.
Posted by Ismaël Mejía <ie...@gmail.com>.
Please don't forget to add this document to the design documents webpage.
On Thu, Jan 31, 2019 at 8:46 PM Alex Amato <aj...@google.com> wrote:
>
> Hello Beam,
>
> Robert Ryan and I have been designing a metric extraction API for Beam. Please take a look at this design, I would love to get more feedback on this to improve the design.
>
> https://s.apache.org/get-metrics-api
>
> The primary goal of this proposal is to offer a simple way to obtain all the metrics for a job. The following issues are addressed:
>
> The current design requires implementing metric querying for every runner+language combination.
>
> Duplication of MetricResult related classes in each language.
>
> The existing MetricResult format only allows querying metrics defined by a namespace, name and step, and does not allow generalized labelling as used by MonitoringInfos.
>
> Enhance Beam’s ability to integration test new metrics
>
>
> Thank for taking a look,
> Alex