You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Lukasz Gajowy (JIRA)" <ji...@apache.org> on 2018/11/20 11:07:00 UTC

[jira] [Work started] (BEAM-6100) Runtime and total bytes metrics are not collected properly

     [ https://issues.apache.org/jira/browse/BEAM-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on BEAM-6100 started by Lukasz Gajowy.
-------------------------------------------
> Runtime and total bytes metrics are not collected properly
> ----------------------------------------------------------
>
>                 Key: BEAM-6100
>                 URL: https://issues.apache.org/jira/browse/BEAM-6100
>             Project: Beam
>          Issue Type: Sub-task
>          Components: testing
>            Reporter: Lukasz Gajowy
>            Assignee: Lukasz Gajowy
>            Priority: Major
>
> Currently, we collect time (distribution) and bytes (counter) metrics from one ParDo (called MetricsMonitor) that is put in pipelines in one, arbitrarily chosen place (usually "in the middle" of pipeline's graph. In some cases, invalid time (or total bytes count) is registered. 
> Taking [this|https://github.com/apache/beam/pull/6987#discussion_r231976671] discussion into account, ideally, we'd like to:
>  - collect runtime by recording time at the root and sink(s) of the pipeline
>  - collect total bytes in a separate ParDo that allows deciding what byte amount do we actually want to collect (now it's coupled to the time-collecting Monitor which is inconvenient).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)