You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Łukasz Gajowy <lg...@apache.org> on 2019/04/19 13:25:56 UTC

New IOIT Dashboards

Hi,

just wanted to announce that we improved the way we collect metrics from
IOIT. Now we use Metrics API for this which allowed us to get more insights
and collect run/read/write time (and possibly other metrics in the future)
separately.

The new dashboards are available here:
https://s.apache.org/io-test-dashboards
I also updated the docs in this PR: https://github.com/apache/beam/pull/8356

Thanks,
Łukasz

Re: New IOIT Dashboards

Posted by Pablo Estrada <pa...@google.com>.
Woah this is excellent. It's very nice to see that the metrics are more
consistent now, and the signal of the tests will be much more useful. Love
it!
Best
-P.

On Fri, Apr 19, 2019 at 1:59 PM Łukasz Gajowy <lu...@gmail.com>
wrote:

> @Kenn Yes, seconds. I added suffixes to widget's legends to clarify.
> @Ankur It's still Dataflow only but we're closer to use Flink for that too
> (we already have the cluster setup in our codebase). Meanwhile, I fixed the
> dashboard title to clarify this too.
>
> Thanks,
> Łukasz
>
> pt., 19 kwi 2019 o 21:35 Ankur Goenka <go...@google.com> napisał(a):
>
>> This looks great!
>> Which runner are we using for the pipeline?
>>
>> On Fri, Apr 19, 2019 at 12:03 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Very cool! I assume times are all in seconds?
>>>
>>> On Fri, Apr 19, 2019 at 6:26 AM Łukasz Gajowy <lg...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> just wanted to announce that we improved the way we collect metrics
>>>> from IOIT. Now we use Metrics API for this which allowed us to get more
>>>> insights and collect run/read/write time (and possibly other metrics in the
>>>> future) separately.
>>>>
>>>> The new dashboards are available here:
>>>> https://s.apache.org/io-test-dashboards
>>>> I also updated the docs in this PR:
>>>> https://github.com/apache/beam/pull/8356
>>>>
>>>> Thanks,
>>>> Łukasz
>>>>
>>>>
>>>>

Re: New IOIT Dashboards

Posted by Łukasz Gajowy <lu...@gmail.com>.
@Kenn Yes, seconds. I added suffixes to widget's legends to clarify.
@Ankur It's still Dataflow only but we're closer to use Flink for that too
(we already have the cluster setup in our codebase). Meanwhile, I fixed the
dashboard title to clarify this too.

Thanks,
Łukasz

pt., 19 kwi 2019 o 21:35 Ankur Goenka <go...@google.com> napisał(a):

> This looks great!
> Which runner are we using for the pipeline?
>
> On Fri, Apr 19, 2019 at 12:03 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Very cool! I assume times are all in seconds?
>>
>> On Fri, Apr 19, 2019 at 6:26 AM Łukasz Gajowy <lg...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> just wanted to announce that we improved the way we collect metrics from
>>> IOIT. Now we use Metrics API for this which allowed us to get more insights
>>> and collect run/read/write time (and possibly other metrics in the future)
>>> separately.
>>>
>>> The new dashboards are available here:
>>> https://s.apache.org/io-test-dashboards
>>> I also updated the docs in this PR:
>>> https://github.com/apache/beam/pull/8356
>>>
>>> Thanks,
>>> Łukasz
>>>
>>>
>>>

Re: New IOIT Dashboards

Posted by Ankur Goenka <go...@google.com>.
This looks great!
Which runner are we using for the pipeline?

On Fri, Apr 19, 2019 at 12:03 PM Kenneth Knowles <ke...@apache.org> wrote:

> Very cool! I assume times are all in seconds?
>
> On Fri, Apr 19, 2019 at 6:26 AM Łukasz Gajowy <lg...@apache.org> wrote:
>
>> Hi,
>>
>> just wanted to announce that we improved the way we collect metrics from
>> IOIT. Now we use Metrics API for this which allowed us to get more insights
>> and collect run/read/write time (and possibly other metrics in the future)
>> separately.
>>
>> The new dashboards are available here:
>> https://s.apache.org/io-test-dashboards
>> I also updated the docs in this PR:
>> https://github.com/apache/beam/pull/8356
>>
>> Thanks,
>> Łukasz
>>
>>
>>

Re: New IOIT Dashboards

Posted by Kenneth Knowles <ke...@apache.org>.
Very cool! I assume times are all in seconds?

On Fri, Apr 19, 2019 at 6:26 AM Łukasz Gajowy <lg...@apache.org> wrote:

> Hi,
>
> just wanted to announce that we improved the way we collect metrics from
> IOIT. Now we use Metrics API for this which allowed us to get more insights
> and collect run/read/write time (and possibly other metrics in the future)
> separately.
>
> The new dashboards are available here:
> https://s.apache.org/io-test-dashboards
> I also updated the docs in this PR:
> https://github.com/apache/beam/pull/8356
>
> Thanks,
> Łukasz
>
>
>