You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2020/05/04 22:33:08 UTC
[Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct link
this time)
Hello,
I have created a proposal for Apache Beam FN API to support Histogram Style
Metrics
<https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit#>.
Which defines a method to collect Histogram style metrics and pass them
over the FN API.
I would love to hear your feedback in order to improve this
proposal, please let me know what you think. Thanks for taking a look :)
Alex
Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct
link this time)
Posted by Robert Bradshaw <ro...@google.com>.
Thanks for the heads up. Though sketches have their advantages, I agree
that it makes sense to stick with bucket counts in the short term for
simplicity and revisit this later.
On Tue, Sep 8, 2020 at 1:08 PM Alex Amato <aj...@google.com> wrote:
> Hi Again, Just reviving this thread to mention that I updated the doc with
> a few sections:
> https://s.apache.org/beam-histogram-metrics
>
>
> Date
>
> Changes
>
> Sept 8, 2020
>
> -
>
> Added alternative section: “Collect Moment Sketch Variables Instead of
> Bucket Counts” (recommend not pursuing, due to opposing trade offs and
> significant implementation/maintenance challenge. But may be worth pursuing
> in a future MonitoringInfo type).
> -
>
> Add distribution variables: min, max, sum, count
> -
>
> Added alternative section: “Update all distribution metrics to be
> Histograms” (recommend not pursuing, update to histogramDistribution on a
> case by case, due to performance concerns).
>
> May 15, 2020
>
> -
>
> Completed review with beam dev list.
>
>
> PTAL and LMK what you think :)
>
> On Wed, May 6, 2020 at 9:58 AM Luke Cwik <lc...@google.com> wrote:
>
>> Thanks Alex, I had some minor comments.
>>
>> On Mon, May 4, 2020 at 4:04 PM Alex Amato <aj...@google.com> wrote:
>>
>>> Thanks Ismaël :). Done
>>>
>>> On Mon, May 4, 2020 at 3:59 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Moving the short link to this thread
>>>> https://s.apache.org/beam-histogram-metrics
>>>>
>>>> Alex can you add this link (and any other of your documents that may
>>>> not be there) to
>>>> https://cwiki.apache.org/confluence/display/BEAM/Design+Documents
>>>>
>>>>
>>>> On Tue, May 5, 2020 at 12:51 AM Pablo Estrada <pa...@google.com>
>>>> wrote:
>>>> >
>>>> > FYI +Boyuan Zhang worked on implementing a histogram metric that was
>>>> performance-optimized into outer space for Python : ) - I don't recall if
>>>> she ended up getting it merged, but it's worth looking at the work. I also
>>>> remember Scott Wegner wrote the metrics for Java.
>>>> >
>>>> > Best
>>>> > -P.
>>>> >
>>>> > On Mon, May 4, 2020 at 3:33 PM Alex Amato <aj...@google.com> wrote:
>>>> >>
>>>> >> Hello,
>>>> >>
>>>> >> I have created a proposal for Apache Beam FN API to support
>>>> Histogram Style Metrics. Which defines a method to collect Histogram style
>>>> metrics and pass them over the FN API.
>>>> >>
>>>> >> I would love to hear your feedback in order to improve this
>>>> proposal, please let me know what you think. Thanks for taking a look :)
>>>> >> Alex
>>>>
>>>
Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct
link this time)
Posted by Alex Amato <aj...@google.com>.
Hi Again, Just reviving this thread to mention that I updated the doc with
a few sections:
https://s.apache.org/beam-histogram-metrics
Date
Changes
Sept 8, 2020
-
Added alternative section: “Collect Moment Sketch Variables Instead of
Bucket Counts” (recommend not pursuing, due to opposing trade offs and
significant implementation/maintenance challenge. But may be worth pursuing
in a future MonitoringInfo type).
-
Add distribution variables: min, max, sum, count
-
Added alternative section: “Update all distribution metrics to be
Histograms” (recommend not pursuing, update to histogramDistribution on a
case by case, due to performance concerns).
May 15, 2020
-
Completed review with beam dev list.
PTAL and LMK what you think :)
On Wed, May 6, 2020 at 9:58 AM Luke Cwik <lc...@google.com> wrote:
> Thanks Alex, I had some minor comments.
>
> On Mon, May 4, 2020 at 4:04 PM Alex Amato <aj...@google.com> wrote:
>
>> Thanks Ismaël :). Done
>>
>> On Mon, May 4, 2020 at 3:59 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Moving the short link to this thread
>>> https://s.apache.org/beam-histogram-metrics
>>>
>>> Alex can you add this link (and any other of your documents that may
>>> not be there) to
>>> https://cwiki.apache.org/confluence/display/BEAM/Design+Documents
>>>
>>>
>>> On Tue, May 5, 2020 at 12:51 AM Pablo Estrada <pa...@google.com>
>>> wrote:
>>> >
>>> > FYI +Boyuan Zhang worked on implementing a histogram metric that was
>>> performance-optimized into outer space for Python : ) - I don't recall if
>>> she ended up getting it merged, but it's worth looking at the work. I also
>>> remember Scott Wegner wrote the metrics for Java.
>>> >
>>> > Best
>>> > -P.
>>> >
>>> > On Mon, May 4, 2020 at 3:33 PM Alex Amato <aj...@google.com> wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> I have created a proposal for Apache Beam FN API to support Histogram
>>> Style Metrics. Which defines a method to collect Histogram style metrics
>>> and pass them over the FN API.
>>> >>
>>> >> I would love to hear your feedback in order to improve this proposal,
>>> please let me know what you think. Thanks for taking a look :)
>>> >> Alex
>>>
>>
Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct
link this time)
Posted by Luke Cwik <lc...@google.com>.
Thanks Alex, I had some minor comments.
On Mon, May 4, 2020 at 4:04 PM Alex Amato <aj...@google.com> wrote:
> Thanks Ismaël :). Done
>
> On Mon, May 4, 2020 at 3:59 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Moving the short link to this thread
>> https://s.apache.org/beam-histogram-metrics
>>
>> Alex can you add this link (and any other of your documents that may
>> not be there) to
>> https://cwiki.apache.org/confluence/display/BEAM/Design+Documents
>>
>>
>> On Tue, May 5, 2020 at 12:51 AM Pablo Estrada <pa...@google.com> wrote:
>> >
>> > FYI +Boyuan Zhang worked on implementing a histogram metric that was
>> performance-optimized into outer space for Python : ) - I don't recall if
>> she ended up getting it merged, but it's worth looking at the work. I also
>> remember Scott Wegner wrote the metrics for Java.
>> >
>> > Best
>> > -P.
>> >
>> > On Mon, May 4, 2020 at 3:33 PM Alex Amato <aj...@google.com> wrote:
>> >>
>> >> Hello,
>> >>
>> >> I have created a proposal for Apache Beam FN API to support Histogram
>> Style Metrics. Which defines a method to collect Histogram style metrics
>> and pass them over the FN API.
>> >>
>> >> I would love to hear your feedback in order to improve this proposal,
>> please let me know what you think. Thanks for taking a look :)
>> >> Alex
>>
>
Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct
link this time)
Posted by Alex Amato <aj...@google.com>.
Thanks Ismaël :). Done
On Mon, May 4, 2020 at 3:59 PM Ismaël Mejía <ie...@gmail.com> wrote:
> Moving the short link to this thread
> https://s.apache.org/beam-histogram-metrics
>
> Alex can you add this link (and any other of your documents that may
> not be there) to
> https://cwiki.apache.org/confluence/display/BEAM/Design+Documents
>
>
> On Tue, May 5, 2020 at 12:51 AM Pablo Estrada <pa...@google.com> wrote:
> >
> > FYI +Boyuan Zhang worked on implementing a histogram metric that was
> performance-optimized into outer space for Python : ) - I don't recall if
> she ended up getting it merged, but it's worth looking at the work. I also
> remember Scott Wegner wrote the metrics for Java.
> >
> > Best
> > -P.
> >
> > On Mon, May 4, 2020 at 3:33 PM Alex Amato <aj...@google.com> wrote:
> >>
> >> Hello,
> >>
> >> I have created a proposal for Apache Beam FN API to support Histogram
> Style Metrics. Which defines a method to collect Histogram style metrics
> and pass them over the FN API.
> >>
> >> I would love to hear your feedback in order to improve this proposal,
> please let me know what you think. Thanks for taking a look :)
> >> Alex
>
Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct
link this time)
Posted by Ismaël Mejía <ie...@gmail.com>.
Moving the short link to this thread
https://s.apache.org/beam-histogram-metrics
Alex can you add this link (and any other of your documents that may
not be there) to
https://cwiki.apache.org/confluence/display/BEAM/Design+Documents
On Tue, May 5, 2020 at 12:51 AM Pablo Estrada <pa...@google.com> wrote:
>
> FYI +Boyuan Zhang worked on implementing a histogram metric that was performance-optimized into outer space for Python : ) - I don't recall if she ended up getting it merged, but it's worth looking at the work. I also remember Scott Wegner wrote the metrics for Java.
>
> Best
> -P.
>
> On Mon, May 4, 2020 at 3:33 PM Alex Amato <aj...@google.com> wrote:
>>
>> Hello,
>>
>> I have created a proposal for Apache Beam FN API to support Histogram Style Metrics. Which defines a method to collect Histogram style metrics and pass them over the FN API.
>>
>> I would love to hear your feedback in order to improve this proposal, please let me know what you think. Thanks for taking a look :)
>> Alex
Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct
link this time)
Posted by Pablo Estrada <pa...@google.com>.
FYI +Boyuan Zhang <bo...@google.com> worked on implementing a histogram
metric that was performance-optimized into outer space for Python : ) - I
don't recall if she ended up getting it merged, but it's worth looking at
the work. I also remember Scott Wegner wrote the metrics for Java.
Best
-P.
On Mon, May 4, 2020 at 3:33 PM Alex Amato <aj...@google.com> wrote:
> Hello,
>
> I have created a proposal for Apache Beam FN API to support Histogram
> Style Metrics
> <https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit#>.
> Which defines a method to collect Histogram style metrics and pass them
> over the FN API.
>
> I would love to hear your feedback in order to improve this
> proposal, please let me know what you think. Thanks for taking a look :)
> Alex
>