Re: [PROPOSAL] Storing, displaying and detecting anomalies in test results

[Kamil Wasilewski <ka...@polidea.com>]

Hi all,

For a while we have been working on the implementation of proposal
presented in this thread. Just to remind what it was about in short words —
we wanted to use Prometheus and Grafana to visualize Beam performance tests
results in addition to detect regressions automatically. A couple of
obstacles forced us to change our approach and that's why I want to share
with you a major proposal revision which you can find here:
https://s.apache.org/test-metrics-storage-corrected.

As usual, we are open to your suggestions and thoughts.

Thanks,
Kamil

On Thu, Sep 5, 2019 at 5:11 PM Kamil Wasilewski <
kamil.wasilewski@polidea.com> wrote:

> We've just started working on this and here comes the first PR:
> https://github.com/apache/beam/pull/9482.
>
> Feel free to share your thoughts and ideas.
>
> Kamil
>
> On Tue, Aug 27, 2019 at 1:19 AM Pablo Estrada <pa...@google.com> wrote:
>
>> Thanks Kamil for bringing this!
>> +Manisha Bhardwaj <ma...@google.com> +Mark Liu <ma...@google.com> have
>> worked on internal benchmarking at Google - would you take a look please?
>>
>> On Fri, Aug 23, 2019 at 3:22 AM Kamil Wasilewski <
>> kamil.wasilewski@polidea.com> wrote:
>>
>>> Hi all,
>>>
>>> Recently we did some research on how to visualize IO performance tests,
>>> Nexmark and Load test results better and how to detect regressions
>>> automatically in an easy way using tools dedicated for the job.
>>>
>>> We'd like to share a proposal with you:
>>> <https://docs.google.com/document/d/1dtHzcp2K3rZihEgNSjFmwN9CF6NbspWhFRaDBdEVNyo/edit?usp=sharing>
>>> https://s.apache.org/test-metrics-storage
>>>
>>>
>>>
>>> Any comments are highly appreciated.
>>>
>>> Thanks,
>>>
>>> Kamil
>>>
>>

Re: [PROPOSAL] Storing, displaying and detecting anomalies in test results

[Kamil Wasilewski <ka...@polidea.com>]

Thanks for spotting this! It should be working fine now.

On Thu, Nov 7, 2019 at 5:40 PM Dan Gazineu <ga...@google.com> wrote:

> Thank you for the update Kamil! Please fix the sharing options in the new
> doc.
>
> On Thu, Nov 7, 2019 at 7:22 AM Kamil Wasilewski <
> kamil.wasilewski@polidea.com> wrote:
>
>> Hi all,
>>
>> For a while we have been working on the implementation of proposal
>> presented in this thread. Just to remind what it was about in short words —
>> we wanted to use Prometheus and Grafana to visualize Beam performance tests
>> results in addition to detect regressions automatically. A couple of
>> obstacles forced us to change our approach and that's why I want to share
>> with you a major proposal revision which you can find here:
>> https://s.apache.org/test-metrics-storage-corrected.
>>
>> As usual, we are open to your suggestions and thoughts.
>>
>> Thanks,
>> Kamil
>>
>> On Thu, Sep 5, 2019 at 5:11 PM Kamil Wasilewski <
>> kamil.wasilewski@polidea.com> wrote:
>>
>>> We've just started working on this and here comes the first PR:
>>> https://github.com/apache/beam/pull/9482.
>>>
>>> Feel free to share your thoughts and ideas.
>>>
>>> Kamil
>>>
>>> On Tue, Aug 27, 2019 at 1:19 AM Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> Thanks Kamil for bringing this!
>>>> +Manisha Bhardwaj <ma...@google.com> +Mark Liu <ma...@google.com> have
>>>> worked on internal benchmarking at Google - would you take a look please?
>>>>
>>>> On Fri, Aug 23, 2019 at 3:22 AM Kamil Wasilewski <
>>>> kamil.wasilewski@polidea.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Recently we did some research on how to visualize IO performance
>>>>> tests, Nexmark and Load test results better and how to detect regressions
>>>>> automatically in an easy way using tools dedicated for the job.
>>>>>
>>>>> We'd like to share a proposal with you:
>>>>> <https://docs.google.com/document/d/1dtHzcp2K3rZihEgNSjFmwN9CF6NbspWhFRaDBdEVNyo/edit?usp=sharing>
>>>>> https://s.apache.org/test-metrics-storage
>>>>>
>>>>>
>>>>>
>>>>> Any comments are highly appreciated.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Kamil
>>>>>
>>>>

Re: [PROPOSAL] Storing, displaying and detecting anomalies in test results

[Dan Gazineu <ga...@google.com>]

Thank you for the update Kamil! Please fix the sharing options in the new
doc.

On Thu, Nov 7, 2019 at 7:22 AM Kamil Wasilewski <
kamil.wasilewski@polidea.com> wrote:

> Hi all,
>
> For a while we have been working on the implementation of proposal
> presented in this thread. Just to remind what it was about in short words —
> we wanted to use Prometheus and Grafana to visualize Beam performance tests
> results in addition to detect regressions automatically. A couple of
> obstacles forced us to change our approach and that's why I want to share
> with you a major proposal revision which you can find here:
> https://s.apache.org/test-metrics-storage-corrected.
>
> As usual, we are open to your suggestions and thoughts.
>
> Thanks,
> Kamil
>
> On Thu, Sep 5, 2019 at 5:11 PM Kamil Wasilewski <
> kamil.wasilewski@polidea.com> wrote:
>
>> We've just started working on this and here comes the first PR:
>> https://github.com/apache/beam/pull/9482.
>>
>> Feel free to share your thoughts and ideas.
>>
>> Kamil
>>
>> On Tue, Aug 27, 2019 at 1:19 AM Pablo Estrada <pa...@google.com> wrote:
>>
>>> Thanks Kamil for bringing this!
>>> +Manisha Bhardwaj <ma...@google.com> +Mark Liu <ma...@google.com> have
>>> worked on internal benchmarking at Google - would you take a look please?
>>>
>>> On Fri, Aug 23, 2019 at 3:22 AM Kamil Wasilewski <
>>> kamil.wasilewski@polidea.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Recently we did some research on how to visualize IO performance tests,
>>>> Nexmark and Load test results better and how to detect regressions
>>>> automatically in an easy way using tools dedicated for the job.
>>>>
>>>> We'd like to share a proposal with you:
>>>> <https://docs.google.com/document/d/1dtHzcp2K3rZihEgNSjFmwN9CF6NbspWhFRaDBdEVNyo/edit?usp=sharing>
>>>> https://s.apache.org/test-metrics-storage
>>>>
>>>>
>>>>
>>>> Any comments are highly appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> Kamil
>>>>
>>>