You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2020/05/06 01:23:30 UTC
[Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Hello,
I created another design document. This time for GCP IO Debuggability
Metrics. Which defines some new metrics to collect in the GCP IO libraries.
This is for monitoring request counts and request latencies.
Please take a look and let me know what you think:
https://s.apache.org/beam-gcp-debuggability
I also sent out a separate design yesterday (
https://s.apache.org/beam-histogram-metrics) which is related as this
document uses a Histogram style metric :).
I would love some feedback to make this feature the best possible :D,
Alex
Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Posted by Alex Amato <aj...@google.com>.
Hi,
Just wanted to mention that I updated this document with one detail
https://s.apache.org/beam-gcp-debuggability
Date
Changes
Sept 8, 2020
-
Clarified that the InstructionRequest/Control Channel will be used in
“Proposal: SDKHs to Report non-bundle metrics.”
May 15, 2020
-
Completed review with beam dev list.
PTAL, and LMK what you think
On Fri, May 15, 2020 at 6:02 PM Alex Amato <aj...@google.com> wrote:
> Thanks everyone. I was able to collect a lot of good feedback from
> everyone who contributed. I am going to wrap it up for now and label the
> design as "Design Finalized (Unimplemented)".
>
> I really believe we have made a much better design than I initially wrote
> up. I couldn't have done it without the help of everyone who offered their
> time, energy and viewpoints. :)
>
> Thanks again, please let me know if you see any major issues with the
> design still. I think I have enough information to begin some
> implementation as soon as I have some time in the coming weeks.
> Alex
>
> https://s.apache.org/beam-gcp-debuggability
> https://s.apache.org/beam-histogram-metrics
>
> On Thu, May 14, 2020 at 5:22 PM Alex Amato <aj...@google.com> wrote:
>
>> Thanks to all who have spent their time on this, there were many great
>> suggestions, just another reminder that tomorrow I will be finalizing the
>> documents, unless there are any major objections left. Please take a look
>> at it if you are interested.
>>
>> I will still welcome feedback at any time :).
>>
>> But I believe we have gathered enough information to produce a good
>> design, which I will start to work on soon.
>> I will begin to build the necessary subset of the new features proposed
>> to support the BigQueryIO metrics use case, proposed.
>> I will likely start with the python SDK first.
>>
>> https://s.apache.org/beam-gcp-debuggability
>> https://s.apache.org/beam-histogram-metrics
>>
>>
>> On Wed, May 13, 2020 at 3:07 PM Alex Amato <aj...@google.com> wrote:
>>
>>> Thanks again for more feedback :). I have iterated on things again. I'll
>>> report back at the end of the week. If there are no major disagreements
>>> still, I'll close the discussion, believe it to be in a good enough state
>>> to start some implementation. But welcome feedback.
>>>
>>> Latest changes are changing the exponential format to allow denser
>>> buckets. Using only two MonitoringInfoSpec now for all of the IOs to use.
>>> Requiring some labels, but allowing optional
>>> ones for specific IOs to provide more contents.
>>>
>>> https://s.apache.org/beam-gcp-debuggability
>>> https://s.apache.org/beam-histogram-metrics
>>>
>>> On Mon, May 11, 2020 at 4:24 PM Alex Amato <aj...@google.com> wrote:
>>>
>>>> Thanks for the great feedback so far :). I've included many new ideas,
>>>> and made some revisions. Both docs have changed a fair bit since the
>>>> initial mail out.
>>>>
>>>> https://s.apache.org/beam-gcp-debuggability
>>>> https://s.apache.org/beam-histogram-metrics
>>>>
>>>> PTAL and let me know what you think, and hopefully we can resolve major
>>>> issues by the end of the week. I'll try to finalize things by then, but of
>>>> course always stay open to your great ideas. :)
>>>>
>>>> On Wed, May 6, 2020 at 6:19 PM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> Thanks everyone so far for taking a look so far :).
>>>>>
>>>>> I am hoping to have this finalize the two reviews by the end of next
>>>>> week, May 15th.
>>>>>
>>>>> I'll continue to follow up on feedback and make changes, and I will
>>>>> add some more mentions to the documents to draw attention
>>>>>
>>>>> https://s.apache.org/beam-gcp-debuggability
>>>>> https://s.apache.org/beam-histogram-metrics
>>>>>
>>>>> On Wed, May 6, 2020 at 10:00 AM Luke Cwik <lc...@google.com> wrote:
>>>>>
>>>>>> Thanks, also took a look and left some comments.
>>>>>>
>>>>>> On Tue, May 5, 2020 at 6:24 PM Alex Amato <aj...@google.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I created another design document. This time for GCP IO
>>>>>>> Debuggability Metrics. Which defines some new metrics to collect in the GCP
>>>>>>> IO libraries. This is for monitoring request counts and request latencies.
>>>>>>>
>>>>>>> Please take a look and let me know what you think:
>>>>>>> https://s.apache.org/beam-gcp-debuggability
>>>>>>>
>>>>>>> I also sent out a separate design yesterday (
>>>>>>> https://s.apache.org/beam-histogram-metrics) which is related as
>>>>>>> this document uses a Histogram style metric :).
>>>>>>>
>>>>>>> I would love some feedback to make this feature the best possible :D,
>>>>>>> Alex
>>>>>>>
>>>>>>
Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Posted by Alex Amato <aj...@google.com>.
Thanks everyone. I was able to collect a lot of good feedback from everyone
who contributed. I am going to wrap it up for now and label the design as
"Design Finalized (Unimplemented)".
I really believe we have made a much better design than I initially wrote
up. I couldn't have done it without the help of everyone who offered their
time, energy and viewpoints. :)
Thanks again, please let me know if you see any major issues with the
design still. I think I have enough information to begin some
implementation as soon as I have some time in the coming weeks.
Alex
https://s.apache.org/beam-gcp-debuggability
https://s.apache.org/beam-histogram-metrics
On Thu, May 14, 2020 at 5:22 PM Alex Amato <aj...@google.com> wrote:
> Thanks to all who have spent their time on this, there were many great
> suggestions, just another reminder that tomorrow I will be finalizing the
> documents, unless there are any major objections left. Please take a look
> at it if you are interested.
>
> I will still welcome feedback at any time :).
>
> But I believe we have gathered enough information to produce a good
> design, which I will start to work on soon.
> I will begin to build the necessary subset of the new features proposed to
> support the BigQueryIO metrics use case, proposed.
> I will likely start with the python SDK first.
>
> https://s.apache.org/beam-gcp-debuggability
> https://s.apache.org/beam-histogram-metrics
>
>
> On Wed, May 13, 2020 at 3:07 PM Alex Amato <aj...@google.com> wrote:
>
>> Thanks again for more feedback :). I have iterated on things again. I'll
>> report back at the end of the week. If there are no major disagreements
>> still, I'll close the discussion, believe it to be in a good enough state
>> to start some implementation. But welcome feedback.
>>
>> Latest changes are changing the exponential format to allow denser
>> buckets. Using only two MonitoringInfoSpec now for all of the IOs to use.
>> Requiring some labels, but allowing optional
>> ones for specific IOs to provide more contents.
>>
>> https://s.apache.org/beam-gcp-debuggability
>> https://s.apache.org/beam-histogram-metrics
>>
>> On Mon, May 11, 2020 at 4:24 PM Alex Amato <aj...@google.com> wrote:
>>
>>> Thanks for the great feedback so far :). I've included many new ideas,
>>> and made some revisions. Both docs have changed a fair bit since the
>>> initial mail out.
>>>
>>> https://s.apache.org/beam-gcp-debuggability
>>> https://s.apache.org/beam-histogram-metrics
>>>
>>> PTAL and let me know what you think, and hopefully we can resolve major
>>> issues by the end of the week. I'll try to finalize things by then, but of
>>> course always stay open to your great ideas. :)
>>>
>>> On Wed, May 6, 2020 at 6:19 PM Alex Amato <aj...@google.com> wrote:
>>>
>>>> Thanks everyone so far for taking a look so far :).
>>>>
>>>> I am hoping to have this finalize the two reviews by the end of next
>>>> week, May 15th.
>>>>
>>>> I'll continue to follow up on feedback and make changes, and I will add
>>>> some more mentions to the documents to draw attention
>>>>
>>>> https://s.apache.org/beam-gcp-debuggability
>>>> https://s.apache.org/beam-histogram-metrics
>>>>
>>>> On Wed, May 6, 2020 at 10:00 AM Luke Cwik <lc...@google.com> wrote:
>>>>
>>>>> Thanks, also took a look and left some comments.
>>>>>
>>>>> On Tue, May 5, 2020 at 6:24 PM Alex Amato <aj...@google.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I created another design document. This time for GCP IO Debuggability
>>>>>> Metrics. Which defines some new metrics to collect in the GCP IO libraries.
>>>>>> This is for monitoring request counts and request latencies.
>>>>>>
>>>>>> Please take a look and let me know what you think:
>>>>>> https://s.apache.org/beam-gcp-debuggability
>>>>>>
>>>>>> I also sent out a separate design yesterday (
>>>>>> https://s.apache.org/beam-histogram-metrics) which is related as
>>>>>> this document uses a Histogram style metric :).
>>>>>>
>>>>>> I would love some feedback to make this feature the best possible :D,
>>>>>> Alex
>>>>>>
>>>>>
Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Posted by Alex Amato <aj...@google.com>.
Thanks to all who have spent their time on this, there were many great
suggestions, just another reminder that tomorrow I will be finalizing the
documents, unless there are any major objections left. Please take a look
at it if you are interested.
I will still welcome feedback at any time :).
But I believe we have gathered enough information to produce a good design,
which I will start to work on soon.
I will begin to build the necessary subset of the new features proposed to
support the BigQueryIO metrics use case, proposed.
I will likely start with the python SDK first.
https://s.apache.org/beam-gcp-debuggability
https://s.apache.org/beam-histogram-metrics
On Wed, May 13, 2020 at 3:07 PM Alex Amato <aj...@google.com> wrote:
> Thanks again for more feedback :). I have iterated on things again. I'll
> report back at the end of the week. If there are no major disagreements
> still, I'll close the discussion, believe it to be in a good enough state
> to start some implementation. But welcome feedback.
>
> Latest changes are changing the exponential format to allow denser
> buckets. Using only two MonitoringInfoSpec now for all of the IOs to use.
> Requiring some labels, but allowing optional
> ones for specific IOs to provide more contents.
>
> https://s.apache.org/beam-gcp-debuggability
> https://s.apache.org/beam-histogram-metrics
>
> On Mon, May 11, 2020 at 4:24 PM Alex Amato <aj...@google.com> wrote:
>
>> Thanks for the great feedback so far :). I've included many new ideas,
>> and made some revisions. Both docs have changed a fair bit since the
>> initial mail out.
>>
>> https://s.apache.org/beam-gcp-debuggability
>> https://s.apache.org/beam-histogram-metrics
>>
>> PTAL and let me know what you think, and hopefully we can resolve major
>> issues by the end of the week. I'll try to finalize things by then, but of
>> course always stay open to your great ideas. :)
>>
>> On Wed, May 6, 2020 at 6:19 PM Alex Amato <aj...@google.com> wrote:
>>
>>> Thanks everyone so far for taking a look so far :).
>>>
>>> I am hoping to have this finalize the two reviews by the end of next
>>> week, May 15th.
>>>
>>> I'll continue to follow up on feedback and make changes, and I will add
>>> some more mentions to the documents to draw attention
>>>
>>> https://s.apache.org/beam-gcp-debuggability
>>> https://s.apache.org/beam-histogram-metrics
>>>
>>> On Wed, May 6, 2020 at 10:00 AM Luke Cwik <lc...@google.com> wrote:
>>>
>>>> Thanks, also took a look and left some comments.
>>>>
>>>> On Tue, May 5, 2020 at 6:24 PM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I created another design document. This time for GCP IO Debuggability
>>>>> Metrics. Which defines some new metrics to collect in the GCP IO libraries.
>>>>> This is for monitoring request counts and request latencies.
>>>>>
>>>>> Please take a look and let me know what you think:
>>>>> https://s.apache.org/beam-gcp-debuggability
>>>>>
>>>>> I also sent out a separate design yesterday (
>>>>> https://s.apache.org/beam-histogram-metrics) which is related as this
>>>>> document uses a Histogram style metric :).
>>>>>
>>>>> I would love some feedback to make this feature the best possible :D,
>>>>> Alex
>>>>>
>>>>
Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Posted by Alex Amato <aj...@google.com>.
Thanks again for more feedback :). I have iterated on things again. I'll
report back at the end of the week. If there are no major disagreements
still, I'll close the discussion, believe it to be in a good enough state
to start some implementation. But welcome feedback.
Latest changes are changing the exponential format to allow denser buckets.
Using only two MonitoringInfoSpec now for all of the IOs to use. Requiring
some labels, but allowing optional
ones for specific IOs to provide more contents.
https://s.apache.org/beam-gcp-debuggability
https://s.apache.org/beam-histogram-metrics
On Mon, May 11, 2020 at 4:24 PM Alex Amato <aj...@google.com> wrote:
> Thanks for the great feedback so far :). I've included many new ideas, and
> made some revisions. Both docs have changed a fair bit since the initial
> mail out.
>
> https://s.apache.org/beam-gcp-debuggability
> https://s.apache.org/beam-histogram-metrics
>
> PTAL and let me know what you think, and hopefully we can resolve major
> issues by the end of the week. I'll try to finalize things by then, but of
> course always stay open to your great ideas. :)
>
> On Wed, May 6, 2020 at 6:19 PM Alex Amato <aj...@google.com> wrote:
>
>> Thanks everyone so far for taking a look so far :).
>>
>> I am hoping to have this finalize the two reviews by the end of next
>> week, May 15th.
>>
>> I'll continue to follow up on feedback and make changes, and I will add
>> some more mentions to the documents to draw attention
>>
>> https://s.apache.org/beam-gcp-debuggability
>> https://s.apache.org/beam-histogram-metrics
>>
>> On Wed, May 6, 2020 at 10:00 AM Luke Cwik <lc...@google.com> wrote:
>>
>>> Thanks, also took a look and left some comments.
>>>
>>> On Tue, May 5, 2020 at 6:24 PM Alex Amato <aj...@google.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I created another design document. This time for GCP IO Debuggability
>>>> Metrics. Which defines some new metrics to collect in the GCP IO libraries.
>>>> This is for monitoring request counts and request latencies.
>>>>
>>>> Please take a look and let me know what you think:
>>>> https://s.apache.org/beam-gcp-debuggability
>>>>
>>>> I also sent out a separate design yesterday (
>>>> https://s.apache.org/beam-histogram-metrics) which is related as this
>>>> document uses a Histogram style metric :).
>>>>
>>>> I would love some feedback to make this feature the best possible :D,
>>>> Alex
>>>>
>>>
Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Posted by Alex Amato <aj...@google.com>.
Thanks for the great feedback so far :). I've included many new ideas, and
made some revisions. Both docs have changed a fair bit since the initial
mail out.
https://s.apache.org/beam-gcp-debuggability
https://s.apache.org/beam-histogram-metrics
PTAL and let me know what you think, and hopefully we can resolve major
issues by the end of the week. I'll try to finalize things by then, but of
course always stay open to your great ideas. :)
On Wed, May 6, 2020 at 6:19 PM Alex Amato <aj...@google.com> wrote:
> Thanks everyone so far for taking a look so far :).
>
> I am hoping to have this finalize the two reviews by the end of next week,
> May 15th.
>
> I'll continue to follow up on feedback and make changes, and I will add
> some more mentions to the documents to draw attention
>
> https://s.apache.org/beam-gcp-debuggability
> https://s.apache.org/beam-histogram-metrics
>
> On Wed, May 6, 2020 at 10:00 AM Luke Cwik <lc...@google.com> wrote:
>
>> Thanks, also took a look and left some comments.
>>
>> On Tue, May 5, 2020 at 6:24 PM Alex Amato <aj...@google.com> wrote:
>>
>>> Hello,
>>>
>>> I created another design document. This time for GCP IO Debuggability
>>> Metrics. Which defines some new metrics to collect in the GCP IO libraries.
>>> This is for monitoring request counts and request latencies.
>>>
>>> Please take a look and let me know what you think:
>>> https://s.apache.org/beam-gcp-debuggability
>>>
>>> I also sent out a separate design yesterday (
>>> https://s.apache.org/beam-histogram-metrics) which is related as this
>>> document uses a Histogram style metric :).
>>>
>>> I would love some feedback to make this feature the best possible :D,
>>> Alex
>>>
>>
Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Posted by Alex Amato <aj...@google.com>.
Thanks everyone so far for taking a look so far :).
I am hoping to have this finalize the two reviews by the end of next week,
May 15th.
I'll continue to follow up on feedback and make changes, and I will add
some more mentions to the documents to draw attention
https://s.apache.org/beam-gcp-debuggability
https://s.apache.org/beam-histogram-metrics
On Wed, May 6, 2020 at 10:00 AM Luke Cwik <lc...@google.com> wrote:
> Thanks, also took a look and left some comments.
>
> On Tue, May 5, 2020 at 6:24 PM Alex Amato <aj...@google.com> wrote:
>
>> Hello,
>>
>> I created another design document. This time for GCP IO Debuggability
>> Metrics. Which defines some new metrics to collect in the GCP IO libraries.
>> This is for monitoring request counts and request latencies.
>>
>> Please take a look and let me know what you think:
>> https://s.apache.org/beam-gcp-debuggability
>>
>> I also sent out a separate design yesterday (
>> https://s.apache.org/beam-histogram-metrics) which is related as this
>> document uses a Histogram style metric :).
>>
>> I would love some feedback to make this feature the best possible :D,
>> Alex
>>
>
Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics
Posted by Luke Cwik <lc...@google.com>.
Thanks, also took a look and left some comments.
On Tue, May 5, 2020 at 6:24 PM Alex Amato <aj...@google.com> wrote:
> Hello,
>
> I created another design document. This time for GCP IO Debuggability
> Metrics. Which defines some new metrics to collect in the GCP IO libraries.
> This is for monitoring request counts and request latencies.
>
> Please take a look and let me know what you think:
> https://s.apache.org/beam-gcp-debuggability
>
> I also sent out a separate design yesterday (
> https://s.apache.org/beam-histogram-metrics) which is related as this
> document uses a Histogram style metric :).
>
> I would love some feedback to make this feature the best possible :D,
> Alex
>